[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487946024 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002327] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.007211] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010006] pid_max: default: 32768 minimum: 301 [ 0.012157] LSM: Security Framework initializing [ 0.013057] Yama: becoming mindful. [ 0.014035] SELinux: Initializing. [ 0.015075] *** VALIDATE selinux *** [ 0.023703] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027400] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028181] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030123] *** VALIDATE tmpfs *** [ 0.032014] *** VALIDATE proc *** [ 0.033217] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.042586] debug: unmapping init [mem 0xffffffff86e59000-0xffffffff86e60fff] [ 0.045310] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046819] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048010] ... bit width: 48 [ 0.049008] ... generic registers: 4 [ 0.050010] ... value mask: 0000ffffffffffff [ 0.051060] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053009] ... event mask: 000000070000000f [ 0.055312] rcu: Hierarchical SRCU implementation. [ 0.057588] smp: Bringing up secondary CPUs ... [ 0.058623] x86: Booting SMP configuration: [ 0.059016] .... node #0, CPUs: #1 #2 #3 [ 0.064119] smp: Brought up 1 node, 4 CPUs [ 0.066014] smpboot: Max logical packages: 1 [ 0.067010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.198035] node 0 deferred pages initialised in 128ms [ 0.203045] devtmpfs: initialized [ 0.204314] x86/mm: Memory block size: 128MB [ 0.209086] gcov: version magic: 0x41383552 [ 0.211641] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.217085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.222865] pinctrl core: initialized pinctrl subsystem [ 0.227184] [ 0.227975] ************************************************************* [ 0.230011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.233010] ** ** [ 0.237013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.240011] ** ** [ 0.243014] ** This means that this kernel is built to expose internal ** [ 0.245016] ** IOMMU data structures, which may compromise security on ** [ 0.248012] ** your system. ** [ 0.250010] ** ** [ 0.253015] ** If you see this message and you are not debugging the ** [ 0.256012] ** kernel, report this immediately to your vendor! ** [ 0.261015] ** ** [ 0.265011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267007] ************************************************************* [ 0.269695] NET: Registered protocol family 16 [ 0.270402] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.272212] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.275056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.279133] cpuidle: using governor menu [ 0.286000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.289831] PCI: Using configuration type 1 for base access [ 0.293162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.302052] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.307121] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.310221] cryptd: max_cpu_qlen set to 1000 [ 0.313810] ACPI: Added _OSI(Module Device) [ 0.316016] ACPI: Added _OSI(Processor Device) [ 0.318018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.322019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.327837] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.337197] ACPI: Interpreter enabled [ 0.340092] ACPI: PM: (supports S0 S3 S4 S5) [ 0.342013] ACPI: Using IOAPIC for interrupt routing [ 0.345221] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.352576] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.364898] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.370042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.376030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.382068] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.393303] acpiphp: Slot [2] registered [ 0.399163] acpiphp: Slot [5] registered [ 0.403183] acpiphp: Slot [6] registered [ 0.406095] acpiphp: Slot [7] registered [ 0.409211] acpiphp: Slot [8] registered [ 0.411199] acpiphp: Slot [9] registered [ 0.418161] acpiphp: Slot [10] registered [ 0.424177] acpiphp: Slot [3] registered [ 0.433143] acpiphp: Slot [4] registered [ 0.436134] acpiphp: Slot [11] registered [ 0.438115] acpiphp: Slot [12] registered [ 0.440194] acpiphp: Slot [13] registered [ 0.442142] acpiphp: Slot [14] registered [ 0.444142] acpiphp: Slot [15] registered [ 0.445120] acpiphp: Slot [16] registered [ 0.447238] acpiphp: Slot [17] registered [ 0.449096] acpiphp: Slot [18] registered [ 0.455321] acpiphp: Slot [19] registered [ 0.456161] acpiphp: Slot [20] registered [ 0.458097] acpiphp: Slot [21] registered [ 0.459090] acpiphp: Slot [22] registered [ 0.460120] acpiphp: Slot [23] registered [ 0.462073] acpiphp: Slot [24] registered [ 0.463062] acpiphp: Slot [25] registered [ 0.464067] acpiphp: Slot [26] registered [ 0.465085] acpiphp: Slot [27] registered [ 0.467084] acpiphp: Slot [28] registered [ 0.468092] acpiphp: Slot [29] registered [ 0.469088] acpiphp: Slot [30] registered [ 0.471074] acpiphp: Slot [31] registered [ 0.472047] PCI host bridge to bus 0000:00 [ 0.473016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.475021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.477018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.478028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.480018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.482022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.484248] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.485926] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.489134] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.497017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.503009] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.505013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.507015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.509020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.512105] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.514728] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.516034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.518717] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.523012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.537012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.541018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.547669] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.554016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.559021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.580016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.598406] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.605065] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.614015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.634074] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.652997] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.665012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.676010] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.716025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.742211] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.761017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.770013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.811017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.826702] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.838012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.851016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.895023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.917000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.932000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.938014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.974014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.995890] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.998395] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.002000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.004420] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.007271] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.013094] iommu: Default domain type: Passthrough [ 1.016464] SCSI subsystem initialized [ 1.018179] ACPI: bus type USB registered [ 1.020085] usbcore: registered new interface driver usbfs [ 1.023074] usbcore: registered new interface driver hub [ 1.025073] usbcore: registered new device driver usb [ 1.027333] pps_core: LinuxPPS API ver. 1 registered [ 1.030012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.033061] PTP clock support registered [ 1.036053] EDAC MC: Ver: 3.0.0 [ 1.039043] PCI: Using ACPI for IRQ routing [ 1.041320] NetLabel: Initializing [ 1.043010] NetLabel: domain hash size = 128 [ 1.046012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.051114] NetLabel: unlabeled traffic allowed by default [ 1.055158] vgaarb: loaded [ 1.058393] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.060014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.073091] clocksource: Switched to clocksource kvm-clock [ 1.230030] VFS: Disk quotas dquot_6.6.0 [ 1.232040] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.234939] *** VALIDATE ramfs *** [ 1.236497] *** VALIDATE hugetlbfs *** [ 1.238290] pnp: PnP ACPI init [ 1.241543] pnp: PnP ACPI: found 6 devices [ 1.269706] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.276510] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.278340] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.279870] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.281748] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.286051] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.290484] NET: Registered protocol family 2 [ 1.294663] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.304593] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.309361] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.316972] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.323290] TCP: Hash tables configured (established 65536 bind 65536) [ 1.326518] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.329939] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.333167] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.336534] NET: Registered protocol family 1 [ 1.339322] RPC: Registered named UNIX socket transport module. [ 1.341539] RPC: Registered udp transport module. [ 1.343316] RPC: Registered tcp transport module. [ 1.345228] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.347882] NET: Registered protocol family 44 [ 1.349772] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.352012] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.354083] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.356326] PCI: CLS 0 bytes, default 64 [ 1.357966] Unpacking initramfs... [ 2.855399] debug: unmapping init [mem 0xffff98947cc54000-0xffff98947ffbffff] [ 2.866085] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.868195] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.874769] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.399376] Initialise system trusted keyrings [ 3.401087] Key type blacklist registered [ 3.403055] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.413574] zbud: loaded [ 3.416419] *** VALIDATE nfs *** [ 3.417874] *** VALIDATE nfs4 *** [ 3.419421] pstore: using deflate compression [ 3.423181] Platform Keyring initialized [ 3.542972] NET: Registered protocol family 38 [ 3.545303] Key type asymmetric registered [ 3.546990] Asymmetric key parser 'x509' registered [ 3.549245] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.552432] io scheduler mq-deadline registered [ 3.554208] io scheduler kyber registered [ 3.555858] io scheduler bfq registered [ 3.557476] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.560198] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.571852] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.574726] ACPI: Power Button [PWRF] [ 3.581241] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.587349] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.613488] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.623327] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.641388] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.673762] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.704490] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.708250] Non-volatile memory driver v1.3 [ 3.709422] Linux agpgart interface v0.103 [ 3.739715] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.742283] vda: detected capacity change from 0 to 74645504 [ 3.759453] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.763331] vdb: detected capacity change from 0 to 1073741824 [ 3.778123] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.780926] vdc: detected capacity change from 0 to 2621440000 [ 3.803505] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.808881] vdd: detected capacity change from 0 to 2621440000 [ 3.837308] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.844319] vde: detected capacity change from 0 to 4294967296 [ 3.866551] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.870259] vdf: detected capacity change from 0 to 4294967296 [ 3.878535] libphy: Fixed MDIO Bus: probed [ 3.886153] usbcore: registered new interface driver usbserial_generic [ 3.888450] usbserial: USB Serial support registered for generic [ 3.894024] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.902446] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.904517] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.908510] mousedev: PS/2 mouse device common for all mice [ 3.912943] rtc_cmos 00:05: RTC can wake from S4 [ 3.915521] rtc_cmos 00:05: registered as rtc0 [ 3.916868] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.919361] intel_pstate: CPU model not supported [ 3.922140] hid: raw HID events driver (C) Jiri Kosina [ 3.923274] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.926128] usbcore: registered new interface driver usbhid [ 3.926134] usbhid: USB HID core driver [ 3.926250] drop_monitor: Initializing network drop monitor service [ 3.926385] Initializing XFRM netlink socket [ 3.927841] NET: Registered protocol family 10 [ 3.942483] Segment Routing with IPv6 [ 3.944057] NET: Registered protocol family 17 [ 3.946942] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.953962] mpls_gso: MPLS GSO support [ 3.959395] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.966175] RAS: Correctable Errors collector initialized. [ 3.972546] AVX version of gcm_enc/dec engaged. [ 3.974080] AES CTR mode by8 optimization enabled [ 4.079430] sched_clock: Marking stable (4079415167, 0)->(5052157715, -972742548) [ 4.084681] registered taskstats version 1 [ 4.086534] Loading compiled-in X.509 certificates [ 4.088380] zswap: loaded using pool lzo/zbud [ 4.109821] Key type big_key registered [ 4.120914] Key type encrypted registered [ 4.122065] ima: No TPM chip found, activating TPM-bypass! [ 4.123566] ima: Allocated hash algorithm: sha1 [ 4.124886] ima: No architecture policies found [ 4.126508] evm: Initialising EVM extended attributes: [ 4.128436] evm: security.selinux [ 4.129413] evm: security.ima [ 4.130323] evm: security.capability [ 4.131456] evm: HMAC attrs: 0x1 [ 4.133446] rtc_cmos 00:05: setting system clock to 2026-07-18 22:14:44 UTC (1784412884) [ 4.140133] debug: unmapping init [mem 0xffffffff87e03000-0xffffffff87ffffff] [ 4.142683] debug: unmapping init [mem 0xffffffff86b82000-0xffffffff86e58fff] [ 4.152062] Write protecting the kernel read-only data: 28672k [ 4.155448] debug: unmapping init [mem 0xffffffff85203000-0xffffffff853fffff] [ 4.157944] debug: unmapping init [mem 0xffffffff85b14000-0xffffffff85bfffff] [ 4.189315] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.196913] systemd[1]: Detected virtualization kvm. [ 4.202416] systemd[1]: Detected architecture x86-64. [ 4.203903] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.229707] systemd[1]: No hostname configured. [ 4.231187] systemd[1]: Set hostname to . [ 4.232752] random: systemd: uninitialized urandom read (16 bytes read) [ 4.234790] systemd[1]: Initializing machine ID from random generator. [ 4.339434] random: ln: uninitialized urandom read (6 bytes read) [ 4.519300] random: systemd: uninitialized urandom read (16 bytes read) [ 4.521570] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.529316] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.538829] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.555954] device-mapper: uevent: version 1.0.3 [ 5.557912] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.931212] random: fast init done [ 7.060296] virtio_net virtio0 ens2: renamed from eth0 [ 7.135105] scsi host0: ata_piix [ 7.214700] scsi host1: ata_piix [ 7.224484] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.238895] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.640854] random: crng init done [ 12.646535] random: 7 urandom warning(s) missed due to ratelimiting [ 14.919798] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.478337] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.071179] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.504981] SELinux: Disabled at runtime. [ 18.572362] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.583788] systemd[1]: Detected virtualization kvm. [ 18.586923] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.638948] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.643334] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.652859] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.657268] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.661519] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.669662] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.676814] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. [ 19.988700] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.655759] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.167895] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.375220] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.550446] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.578699] EDAC sbridge: Ver: 1.1.2 [ 26.313040] Key type dns_resolver registered [ 26.776976] NFS: Registering the id_resolver key type [ 26.779287] Key type id_resolver registered [ 26.781075] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg328-server login: [ 61.951446] libcfs: loading out-of-tree module taints kernel. [ 61.975844] Key type ._llcrypt registered [ 61.977681] Key type .llcrypt registered [ 62.054403] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 70.838956] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 71.481627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 71.489284] alg: No test for adler32 (adler32-zlib) [ 72.549925] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 72.894677] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 74.520153] Key type lgssc registered [ 75.115657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 82.139861] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 99.530448] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 104.882530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 104.893725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 106.010125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 106.028513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 106.074211] Lustre: lustre-MDT0000: new disk, initializing [ 106.110174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 106.120540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 107.744904] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.956359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 114.007715] Lustre: 6479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 114.033837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 114.036435] Lustre: Skipped 1 previous similar message [ 114.085592] Lustre: lustre-MDT0001: new disk, initializing [ 114.117231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 114.130511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 114.138544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 115.768705] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 118.316522] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 122.176504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 122.277580] Lustre: lustre-OST0000: new disk, initializing [ 122.279149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 122.281574] Lustre: 8415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 122.302610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 124.394839] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 129.520800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 129.526054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 129.560527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 130.463534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 130.517620] Lustre: lustre-OST0001: new disk, initializing [ 130.519633] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 130.522573] Lustre: 9485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 130.560413] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 132.980340] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 138.756347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 138.760454] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 138.790147] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 139.604843] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 144.562362] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 146.785746] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing check_logdir /tmp/testlogs/ [ 148.643550] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing yml_node [ 150.456137] Lustre: DEBUG MARKER: Client: 2.17.54.173 [ 151.455588] Lustre: DEBUG MARKER: MDS: 2.17.54.173 [ 152.476245] Lustre: DEBUG MARKER: OSS: 2.17.54.173 [ 153.149414] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Jul 18 18:17:12 EDT 2026 [ 159.733432] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 160.319366] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 174.560674] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 174.561303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 174.566546] Lustre: Skipped 2 previous similar messages [ 174.569069] Lustre: Skipped 1 previous similar message [ 178.344669] Lustre: server umount lustre-MDT0000 complete [ 179.831102] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413060 with bad export cookie 1742144318969603613 [ 179.833047] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 179.837670] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 184.800579] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 184.801865] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 184.808033] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 184.813203] Lustre: Skipped 1 previous similar message [ 184.819831] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 184.822330] Lustre: Skipped 3 previous similar messages [ 186.030805] Lustre: server umount lustre-MDT0001 complete [ 193.744150] Lustre: server umount lustre-OST0000 complete [ 201.708405] Lustre: server umount lustre-OST0001 complete [ 204.736788] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 209.087367] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 226.296557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.423348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 226.444374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 226.505351] Lustre: lustre-MDT0000: new disk, initializing [ 226.563708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.574544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 228.506805] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 233.831427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.898127] Lustre: 15029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 234.053649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 235.796595] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 240.270930] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 242.637423] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 245.452754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.567814] Lustre: lustre-OST0000: new disk, initializing [ 245.570124] Lustre: Skipped 1 previous similar message [ 245.572300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.574650] Lustre: Skipped 2 previous similar messages [ 245.577965] Lustre: 16996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.613559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 245.617567] Lustre: Skipped 1 previous similar message [ 246.599389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 246.603984] Lustre: Skipped 1 previous similar message [ 246.606155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 246.648358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.804349] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 252.066819] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 253.669869] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 253.757677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 255.403375] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 255.497239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 256.993432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 256.996072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 256.996799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 256.997040] LustreError: Skipped 1 previous similar message [ 257.002743] Lustre: Skipped 2 previous similar messages [ 262.117087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 262.120965] Lustre: Skipped 2 previous similar messages [ 262.376849] Lustre: server umount lustre-OST0000 complete [ 264.680766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 264.684254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 267.234976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 267.242412] Lustre: Skipped 2 previous similar messages [ 270.130753] Lustre: server umount lustre-MDT0000 complete [ 271.715505] LustreError: 15825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413152 with bad export cookie 1742144318969605146 [ 271.718787] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 271.720430] LustreError: 15825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 271.866169] Lustre: server umount lustre-MDT0001 complete [ 273.308724] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 18:19:13 (1784413153) [ 275.940830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 276.135267] LustreError: 19259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 276.171600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 277.717714] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 281.022972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 282.749399] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 285.579183] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 287.874224] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 290.699309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 293.296510] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 296.256346] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 306.145983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.151151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 306.155127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 306.156299] Lustre: Skipped 2 previous similar messages [ 310.501710] Lustre: server umount lustre-OST0000 complete [ 316.386136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.396913] Lustre: Skipped 1 previous similar message [ 318.395745] Lustre: server umount lustre-MDT0000 complete [ 320.028643] LustreError: 19240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413200 with bad export cookie 1742144318969606028 [ 320.030280] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.036154] LustreError: 19240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 320.188551] Lustre: server umount lustre-MDT0001 complete [ 324.727742] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 326.072866] Key type lgssc unregistered [ 326.219217] LNet: 22787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.226572] LNetError: 22787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 326.239491] LNet: Removed LNI 192.168.203.128@tcp [ 326.684224] Key type .llcrypt unregistered [ 326.686402] Key type ._llcrypt unregistered [ 337.352929] Key type ._llcrypt registered [ 337.354310] Key type .llcrypt registered [ 337.403527] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 337.878910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 337.888339] alg: No test for adler32 (adler32-zlib) [ 338.778289] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 338.894302] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 340.488216] Key type lgssc registered [ 340.955888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 344.270199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 344.280868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.472799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 347.276426] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 351.406657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 351.585143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 353.423615] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 356.975420] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 359.368738] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 362.399937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 362.585125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 365.249449] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 368.780141] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 369.773281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 381.410719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 381.413700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 381.419221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 381.422186] Lustre: Skipped 1 previous similar message [ 382.805622] Lustre: server umount lustre-OST0000 complete [ 386.395438] Lustre: server umount lustre-MDT0000 complete [ 386.530281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.531187] LustreError: 24171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.535986] Lustre: Skipped 2 previous similar messages [ 386.543182] LustreError: 24171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 387.833292] LustreError: 24158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413268 with bad export cookie 9423181240165460640 [ 387.837143] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 387.838493] LustreError: 24158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 387.969122] Lustre: server umount lustre-MDT0001 complete [ 392.150709] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 393.348095] Key type lgssc unregistered [ 393.485983] LNet: 28150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.492689] LNetError: 28150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 393.504969] LNet: Removed LNI 192.168.203.128@tcp [ 393.907183] Key type .llcrypt unregistered [ 393.908581] Key type ._llcrypt unregistered [ 404.364293] Key type ._llcrypt registered [ 404.366646] Key type .llcrypt registered [ 404.419404] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 404.837375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 404.891124] alg: No test for adler32 (adler32-zlib) [ 405.776825] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 405.887142] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 407.488704] Key type lgssc registered [ 407.974089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 411.668686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 411.681749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.887935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.633664] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 418.470279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.658905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 420.509304] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 423.647992] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 425.974406] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 431.843619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 432.003147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.720497] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 437.827810] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 438.184893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 448.481901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 448.484840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.487796] LustreError: Skipped 1 previous similar message [ 448.496853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 448.506512] Lustre: Skipped 1 previous similar message [ 451.736320] Lustre: server umount lustre-OST0000 complete [ 453.604311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.606909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 453.610719] Lustre: Skipped 2 previous similar messages [ 453.620358] Lustre: Skipped 1 previous similar message [ 458.721509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 458.726378] Lustre: Skipped 1 previous similar message [ 459.388820] Lustre: server umount lustre-MDT0000 complete [ 460.956255] LustreError: 29521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413341 with bad export cookie 540027563038190701 [ 460.959033] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 460.963602] LustreError: 29521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 461.076882] Lustre: server umount lustre-MDT0001 complete [ 465.514815] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 466.721058] Key type lgssc unregistered [ 466.858619] LNet: 33412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 466.863261] LNetError: 33412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 466.870457] LNet: Removed LNI 192.168.203.128@tcp [ 467.240284] Key type .llcrypt unregistered [ 467.241851] Key type ._llcrypt unregistered [ 477.299435] Key type ._llcrypt registered [ 477.300836] Key type .llcrypt registered [ 477.342827] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 477.803065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 477.811055] alg: No test for adler32 (adler32-zlib) [ 478.683225] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 478.784292] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 480.384163] Key type lgssc registered [ 480.922200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 484.710856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 484.721468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.940104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.719597] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 491.529476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.691841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 493.577535] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 497.013468] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 499.512504] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 502.564334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 502.732471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.445319] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 508.842263] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 510.955336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 526.305876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 526.314180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.321956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 527.331096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 527.334127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.334197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.002214] Lustre: server umount lustre-OST0000 complete [ 532.453302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.456323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 532.461977] Lustre: Skipped 1 previous similar message [ 532.464187] Lustre: Skipped 1 previous similar message [ 537.571288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 537.574194] Lustre: Skipped 2 previous similar messages [ 542.689198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 542.692248] Lustre: Skipped 1 previous similar message [ 545.248114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 545.334567] Lustre: server umount lustre-MDT0000 complete [ 546.786619] LustreError: 34821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.796907] LustreError: 34821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 547.102255] LustreError: 34782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413427 with bad export cookie 15883494921524898129 [ 547.106687] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.110203] LustreError: 34782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 547.293796] Lustre: server umount lustre-MDT0001 complete [ 551.821863] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 553.161172] Key type lgssc unregistered [ 553.309774] LNet: 38352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 553.314891] LNetError: 38352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 553.328792] LNet: Removed LNI 192.168.203.128@tcp [ 553.733815] Key type .llcrypt unregistered [ 553.735620] Key type ._llcrypt unregistered [ 564.276943] Key type ._llcrypt registered [ 564.278133] Key type .llcrypt registered [ 564.322819] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 564.710210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 564.751047] alg: No test for adler32 (adler32-zlib) [ 565.635564] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 565.745562] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 567.336173] Key type lgssc registered [ 567.812086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 570.872129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 570.883216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.059173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.685791] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 577.048218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.228185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 578.874380] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 581.709323] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 583.886734] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 586.466712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 586.615102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 588.965964] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 592.038722] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 596.843142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 602.086564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 602.087076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.091220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 602.096228] Lustre: Skipped 1 previous similar message [ 602.100643] Lustre: Skipped 1 previous similar message [ 603.806517] Lustre: lustre-OST0000: Not available for connect from 192.168.203.28@tcp (stopping) [ 607.203789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 607.208920] Lustre: Skipped 1 previous similar message [ 612.324960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 612.328453] Lustre: Skipped 2 previous similar messages [ 615.392112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 615.474451] Lustre: server umount lustre-OST0000 complete [ 629.350134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.350751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 629.355966] Lustre: Skipped 1 previous similar message [ 629.358945] Lustre: Skipped 1 previous similar message [ 635.508485] Lustre: server umount lustre-MDT0000 complete [ 636.994191] LustreError: 39722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413517 with bad export cookie 15583022276888715552 [ 636.997330] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.999160] LustreError: 39722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 643.195410] Lustre: server umount lustre-MDT0001 complete [ 647.464954] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 648.617908] Key type lgssc unregistered [ 648.767261] LNet: 43446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.771754] LNetError: 43446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 648.786716] LNet: Removed LNI 192.168.203.128@tcp [ 649.147419] Key type .llcrypt unregistered [ 649.148882] Key type ._llcrypt unregistered [ 659.275578] Key type ._llcrypt registered [ 659.277050] Key type .llcrypt registered [ 659.321541] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 659.845179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.861239] alg: No test for adler32 (adler32-zlib) [ 660.749509] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 660.857963] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 662.456511] Key type lgssc registered [ 662.899225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 665.912161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 665.924834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.110398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.767848] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 672.393559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.556372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 674.364984] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 677.794225] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 680.186130] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 683.099615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 683.266042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 686.013199] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 689.333938] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 692.458164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 702.946651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.948495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.952988] Lustre: Skipped 1 previous similar message [ 706.202152] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.28@tcp (stopping) [ 706.205898] Lustre: Skipped 2 previous similar messages [ 708.069863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.073636] Lustre: Skipped 2 previous similar messages [ 708.387833] Lustre: server umount lustre-MDT0000 complete [ 709.909621] LustreError: 44817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413590 with bad export cookie 6130400078870278587 [ 709.913780] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 709.915742] LustreError: 44817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 711.314402] LustreError: 44831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.325296] LustreError: 44831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 711.326420] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.28@tcp (stopping) [ 713.185197] LustreError: 44832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.185478] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.197070] Lustre: Skipped 1 previous similar message [ 714.208733] LustreError: 44830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.098895] Lustre: server umount lustre-MDT0001 complete [ 727.126869] LustreError: 47638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.28@tcp arrived at 1784413607 with bad export cookie 6130400078870279231 [ 727.131772] LustreError: 47638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 742.368113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 742.436789] Lustre: server umount lustre-OST0000 complete [ 749.058170] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 750.364468] Key type lgssc unregistered [ 750.521764] LNet: 48717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.525436] LNetError: 48717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.540351] LNet: Removed LNI 192.168.203.128@tcp [ 750.926514] Key type .llcrypt unregistered [ 750.928415] Key type ._llcrypt unregistered [ 762.134105] Key type ._llcrypt registered [ 762.135529] Key type .llcrypt registered [ 762.183315] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 762.649621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 762.669026] alg: No test for adler32 (adler32-zlib) [ 763.537775] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 763.642886] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 765.240154] Key type lgssc registered [ 765.713967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 768.908276] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 768.916481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 784.353781] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 789.472345] LustreError: 50082:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 816.096263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 818.411035] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 821.078399] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 985.245258] Lustre: server umount lustre-OST0000 complete [ 990.938682] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 992.039689] Key type lgssc unregistered [ 992.181461] LNet: 51757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.184722] LNetError: 51757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.197658] LNet: Removed LNI 192.168.203.128@tcp [ 992.522599] Key type .llcrypt unregistered [ 992.523710] Key type ._llcrypt unregistered [ 1001.512529] Key type ._llcrypt registered [ 1001.513886] Key type .llcrypt registered [ 1001.551116] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1001.983731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1002.007660] alg: No test for adler32 (adler32-zlib) [ 1002.857291] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1002.942647] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1004.520132] Key type lgssc registered [ 1004.866151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1007.312715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1007.318520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.469455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1009.825630] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1012.598304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.730519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1014.129856] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1016.441164] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1018.209447] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1020.416701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1020.542257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.437066] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1024.923431] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1029.091076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.091428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1029.095571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1031.824908] Lustre: server umount lustre-OST0000 complete [ 1033.184803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1033.187595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.192248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1034.209101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1034.209847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.189345] Lustre: server umount lustre-MDT0000 complete [ 1039.328984] LustreError: 53145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.335141] LustreError: 53145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1040.468993] LustreError: 53128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413920 with bad export cookie 2741632421257741178 [ 1040.471046] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.473881] LustreError: 53128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1040.592814] Lustre: server umount lustre-MDT0001 complete [ 1044.195349] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1045.268568] Key type lgssc unregistered [ 1045.405944] LNet: 56670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.410677] LNetError: 56670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.418469] LNet: Removed LNI 192.168.203.128@tcp [ 1045.776083] Key type .llcrypt unregistered [ 1045.777434] Key type ._llcrypt unregistered [ 1054.868160] Key type ._llcrypt registered [ 1054.869361] Key type .llcrypt registered [ 1054.909662] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1055.297693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1055.338048] alg: No test for adler32 (adler32-zlib) [ 1056.224966] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1056.333988] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1057.920175] Key type lgssc registered [ 1058.310626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1060.974694] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1060.980768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1076.384357] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1081.504295] LustreError: 58032:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 1108.128244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.063270] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1112.579247] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1115.036000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.240236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.268711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1116.765716] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1119.979097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.124699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1121.636886] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1124.183722] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1126.009565] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1128.418847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.420057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.423685] Lustre: Skipped 1 previous similar message [ 1133.537379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1133.539626] Lustre: Skipped 2 previous similar messages [ 1138.657750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1138.660136] Lustre: Skipped 1 previous similar message [ 1141.216113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1141.270267] Lustre: server umount lustre-OST0000 complete [ 1155.208237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.213818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1155.216205] Lustre: Skipped 1 previous similar message [ 1160.673498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1160.676153] Lustre: Skipped 1 previous similar message [ 1161.425573] Lustre: server umount lustre-MDT0000 complete [ 1162.618102] LustreError: 58038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414042 with bad export cookie 3557230501228716285 [ 1162.622050] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1162.623240] LustreError: 58038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1168.815183] Lustre: server umount lustre-MDT0001 complete [ 1172.167583] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1173.089454] Key type lgssc unregistered [ 1173.203363] LNet: 61774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.205977] LNetError: 61774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1173.213257] LNet: Removed LNI 192.168.203.128@tcp [ 1173.481471] Key type .llcrypt unregistered [ 1173.482601] Key type ._llcrypt unregistered [ 1181.515524] Key type ._llcrypt registered [ 1181.516461] Key type .llcrypt registered [ 1181.549131] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1181.881272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1181.894328] alg: No test for adler32 (adler32-zlib) [ 1182.733170] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1182.814034] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1184.392151] Key type lgssc registered [ 1184.707505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1186.985387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1186.991190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.114680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1189.286509] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1191.878993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.988700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1193.165754] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1195.328677] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1196.949477] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1199.010964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1199.114908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.944195] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1203.229691] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1213.479942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1217.505424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1217.507883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.513465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1218.529599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1218.530727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1218.532824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.772496] Lustre: server umount lustre-OST0000 complete [ 1223.649370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.650026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.654029] Lustre: Skipped 1 previous similar message [ 1223.655684] Lustre: Skipped 1 previous similar message [ 1228.769103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1228.771238] Lustre: Skipped 2 previous similar messages [ 1229.085028] Lustre: server umount lustre-MDT0000 complete [ 1230.304166] LustreError: 63143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414110 with bad export cookie 4661900384309693975 [ 1230.306703] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.308596] LustreError: 63143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1230.425831] Lustre: server umount lustre-MDT0001 complete [ 1233.951495] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1234.975400] Key type lgssc unregistered [ 1235.098319] LNet: 66712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.100715] LNetError: 66712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1235.109306] LNet: Removed LNI 192.168.203.128@tcp [ 1235.417070] Key type .llcrypt unregistered [ 1235.418369] Key type ._llcrypt unregistered [ 1244.665101] Key type ._llcrypt registered [ 1244.666274] Key type .llcrypt registered [ 1244.700404] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1245.010228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1245.055651] alg: No test for adler32 (adler32-zlib) [ 1245.900433] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1245.982153] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1247.560137] Key type lgssc registered [ 1247.904078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1250.605954] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 18:35:30 (1784414130) [ 1252.732253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1252.738287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.873151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.105818] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1257.697304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.809088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1258.977470] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1261.121837] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1262.752095] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1264.899444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1265.026202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.984597] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1269.472052] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1279.398683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1282.175571] Lustre: Failing over lustre-MDT0000 [ 1282.357239] Lustre: server umount lustre-MDT0000 complete [ 1283.553355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1283.556268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.560574] LustreError: 69118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1283.565891] LustreError: 69118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1284.577087] LustreError: 68389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.578728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.582739] LustreError: 68389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1284.587445] Lustre: Skipped 1 previous similar message [ 1285.149486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.183294] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1285.235494] Lustre: *** cfs_fail_loc=135, val=0*** [ 1285.273078] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.28@tcp (not set up) [ 1285.273227] LustreError: 71494:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1285.278398] LustreError: 71494:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -2 [ 1285.282495] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1285.282495] [ 1285.285900] LustreError: MGC192.168.203.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1285.290068] LustreError: 71483:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1285.293507] LustreError: 71483:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1285.297984] LustreError: 71483:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1285.301040] LustreError: 71483:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1285.311468] Lustre: server umount lustre-MDT0000 complete [ 1285.312675] LustreError: 71483:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1287.289814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.326560] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1287.401339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.419684] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1288.568444] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1290.381963] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1292.780126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1292.790166] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1292.808225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1293.792776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1293.794968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.799766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1297.891154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1297.892254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1297.896368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.722118] Lustre: server umount lustre-OST0000 complete [ 1303.010792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1303.011075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.018983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1303.020830] Lustre: Skipped 2 previous similar messages [ 1308.129231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1308.131656] Lustre: Skipped 1 previous similar message [ 1315.296071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1315.346946] Lustre: server umount lustre-MDT0000 complete [ 1316.529160] LustreError: 68370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414196 with bad export cookie 2499712731590614108 [ 1316.533098] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1316.533846] LustreError: 68370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1316.640941] Lustre: server umount lustre-MDT0001 complete [ 1319.986112] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1320.900305] Key type lgssc unregistered [ 1321.016337] LNet: 73270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.018492] LNetError: 73270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.025279] LNet: Removed LNI 192.168.203.128@tcp [ 1321.308924] Key type .llcrypt unregistered [ 1321.309753] Key type ._llcrypt unregistered [ 1329.889378] Key type ._llcrypt registered [ 1329.890570] Key type .llcrypt registered [ 1329.929663] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1330.375436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1330.390616] alg: No test for adler32 (adler32-zlib) [ 1331.249350] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1331.339697] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1332.920157] Key type lgssc registered [ 1333.287541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1335.763458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1335.770130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1336.906514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.202766] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1340.994786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.126301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1342.454774] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1344.605307] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1346.160816] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1348.202992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1348.307699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.001436] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1352.253793] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1362.667323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1369.746083] Lustre: Failing over lustre-MDT0000 [ 1369.968153] Lustre: server umount lustre-MDT0000 complete [ 1372.129280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1372.132515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1372.139147] LustreError: 74658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.145236] LustreError: 74658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1372.800688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.820664] LustreError: 74653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.837907] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1372.889299] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1372.935640] LustreError: 77764:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1372.937821] LustreError: 77764:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -9 [ 1372.940471] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1372.940471] [ 1372.943232] LustreError: MGC192.168.203.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1372.948591] LustreError: 77753:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1372.950872] LustreError: 77753:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 1372.954125] LustreError: 77753:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1372.955863] LustreError: 77753:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1372.964910] Lustre: server umount lustre-MDT0000 complete [ 1372.966247] LustreError: 77753:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1373.154512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1373.159766] Lustre: Skipped 1 previous similar message [ 1374.997485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1375.030262] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1375.121689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.155613] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1376.318962] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1377.935372] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1380.328266] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1380.335085] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1380.350930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1382.368798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1382.371676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.375287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1385.441637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1385.443120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1385.444264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1387.273060] Lustre: server umount lustre-OST0000 complete [ 1390.561861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.562098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1390.562270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.562273] Lustre: Skipped 1 previous similar message [ 1390.567140] Lustre: Skipped 1 previous similar message [ 1394.452066] Lustre: server umount lustre-MDT0000 complete [ 1395.570071] LustreError: 74639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414275 with bad export cookie 5718090403305238679 [ 1395.572765] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1395.572987] LustreError: 74639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1395.673402] Lustre: server umount lustre-MDT0001 complete [ 1399.087287] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1399.993265] Key type lgssc unregistered [ 1400.117269] LNet: 79540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1400.119502] LNetError: 79540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1400.125253] LNet: Removed LNI 192.168.203.128@tcp [ 1400.424259] Key type .llcrypt unregistered [ 1400.425310] Key type ._llcrypt unregistered [ 1404.190889] Key type ._llcrypt registered [ 1404.191855] Key type .llcrypt registered [ 1404.222200] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1404.302338] Key type .llcrypt unregistered [ 1404.303531] Key type ._llcrypt unregistered [ 1412.353407] Key type ._llcrypt registered [ 1412.355047] Key type .llcrypt registered [ 1412.390576] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1412.754631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1412.795374] alg: No test for adler32 (adler32-zlib) [ 1413.644554] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1413.727060] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1415.304154] Key type lgssc registered [ 1415.627809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1418.087160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1418.093671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1419.226886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.478648] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1423.298233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1423.424893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1424.764953] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1427.143450] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1428.845461] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1430.956485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1431.052167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.800878] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1434.990718] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1444.391189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1448.416883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1448.419055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.424604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.444350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1449.444485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.446899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1454.296718] Lustre: server umount lustre-OST0000 complete [ 1459.168748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1459.171110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1459.175382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1459.177656] Lustre: Skipped 1 previous similar message [ 1461.716249] Lustre: server umount lustre-MDT0000 complete [ 1462.928308] LustreError: 81441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414343 with bad export cookie 8407254269480454759 [ 1462.931728] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1462.932748] LustreError: 81441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1463.043304] Lustre: server umount lustre-MDT0001 complete [ 1466.431559] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1467.407148] Key type lgssc unregistered [ 1467.525785] LNet: 84971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.528109] LNetError: 84971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.537248] LNet: Removed LNI 192.168.203.128@tcp [ 1467.828874] Key type .llcrypt unregistered [ 1467.830387] Key type ._llcrypt unregistered [ 1475.984220] Key type ._llcrypt registered [ 1475.985340] Key type .llcrypt registered [ 1476.019986] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1476.375749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1476.395226] alg: No test for adler32 (adler32-zlib) [ 1477.244200] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1477.339414] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1478.928176] Key type lgssc registered [ 1479.289302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1481.674366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1481.679803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.823675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.147876] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1486.874275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.009637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1488.346950] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1490.661289] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1492.435219] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1494.649186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1494.775304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.622297] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1498.875549] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1508.071165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1512.416827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1512.418872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.422452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1513.443296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1513.443688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.443708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1517.772369] Lustre: server umount lustre-OST0000 complete [ 1523.168840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1523.171257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.176050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.178748] Lustre: Skipped 1 previous similar message [ 1525.141068] Lustre: server umount lustre-MDT0000 complete [ 1526.375408] LustreError: 86342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414406 with bad export cookie 9617790933977594004 [ 1526.380504] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1526.490203] Lustre: server umount lustre-MDT0001 complete [ 1529.858718] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1530.852096] Key type lgssc unregistered [ 1530.975367] LNet: 89910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.977649] LNetError: 89910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.989290] LNet: Removed LNI 192.168.203.128@tcp [ 1531.291031] Key type .llcrypt unregistered [ 1531.292631] Key type ._llcrypt unregistered [ 1539.651109] Key type ._llcrypt registered [ 1539.652079] Key type .llcrypt registered [ 1539.688513] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1540.017872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1540.049486] alg: No test for adler32 (adler32-zlib) [ 1540.910344] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1540.995758] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1542.568197] Key type lgssc registered [ 1542.859377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1545.219129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1545.224488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1546.367408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.717798] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1550.458620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1550.576759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1551.911934] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1554.395819] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1556.294175] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1558.587213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1558.705710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.655600] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1563.110177] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1570.980060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1580.512852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1580.514848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1580.518661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1581.026288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1581.028376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.031982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1582.738357] Lustre: server umount lustre-OST0000 complete [ 1586.657770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1586.657840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1586.659218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1586.659224] Lustre: Skipped 1 previous similar message [ 1586.667254] Lustre: Skipped 1 previous similar message [ 1590.106465] Lustre: server umount lustre-MDT0000 complete [ 1591.338327] LustreError: 91281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414471 with bad export cookie 3730390854361913875 [ 1591.341694] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1591.342153] LustreError: 91281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1591.458955] Lustre: server umount lustre-MDT0001 complete [ 1595.011590] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1596.055497] Key type lgssc unregistered [ 1596.178491] LNet: 94850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.182756] LNetError: 94850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.193335] LNet: Removed LNI 192.168.203.128@tcp [ 1596.528200] Key type .llcrypt unregistered [ 1596.529401] Key type ._llcrypt unregistered [ 1605.335847] Key type ._llcrypt registered [ 1605.336955] Key type .llcrypt registered [ 1605.381465] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1605.777035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1605.807452] alg: No test for adler32 (adler32-zlib) [ 1606.666470] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1606.763590] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1608.344176] Key type lgssc registered [ 1608.753648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1611.403236] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1611.409647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1626.848404] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1631.968332] LustreError: 96221:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 1658.592191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.543792] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1662.891666] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1664.661292] Lustre: server umount lustre-OST0000 complete [ 1667.173922] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:42:27 (1784414547) [ 1669.268989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1669.394132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.530136] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1673.124456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1673.233675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1674.464640] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1676.653719] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1678.362377] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1680.477337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1680.583797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.271589] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1684.597353] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1693.924684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1699.299497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.299815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1699.304625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1699.309854] Lustre: Skipped 1 previous similar message [ 1700.757500] Lustre: server umount lustre-OST0000 complete [ 1703.905104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1703.907556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.910934] Lustre: Skipped 1 previous similar message [ 1703.912782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1704.216941] Lustre: server umount lustre-MDT0000 complete [ 1704.416968] LustreError: 97858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1704.423372] LustreError: 97858:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1705.508395] LustreError: 97839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414585 with bad export cookie 8785833858715876238 [ 1705.511748] LustreError: 97839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1705.512177] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1705.634989] Lustre: server umount lustre-MDT0001 complete [ 1709.317348] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1710.351391] Key type lgssc unregistered [ 1710.487418] LNet: 101564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.491193] LNetError: 101564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.499330] LNet: Removed LNI 192.168.203.128@tcp [ 1710.818533] Key type .llcrypt unregistered [ 1710.819895] Key type ._llcrypt unregistered [ 1725.810771] Key type ._llcrypt registered [ 1725.812140] Key type .llcrypt registered [ 1725.861334] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1726.378939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1726.391796] alg: No test for adler32 (adler32-zlib) [ 1727.254156] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1727.347938] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1728.928158] Key type lgssc registered [ 1729.318877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1731.886096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1731.891497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.018566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.257014] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1737.123548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.250399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1738.556616] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1741.083758] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1742.976417] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1745.216286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1745.341706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.313182] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1749.760281] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1757.672567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1767.909162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1767.909792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.911418] LustreError: Skipped 1 previous similar message [ 1767.911964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1767.916519] Lustre: Skipped 1 previous similar message [ 1767.921825] Lustre: Skipped 1 previous similar message [ 1769.098604] Lustre: server umount lustre-OST0000 complete [ 1773.026548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1773.026959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.028262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.035933] Lustre: Skipped 1 previous similar message [ 1776.485449] Lustre: server umount lustre-MDT0000 complete [ 1777.732242] LustreError: 103705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414658 with bad export cookie 9847832429903382116 [ 1777.735439] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1777.737564] LustreError: 103705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1777.855690] Lustre: server umount lustre-MDT0001 complete [ 1781.614259] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1782.621931] Key type lgssc unregistered [ 1782.744374] LNet: 107277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.747273] LNetError: 107277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.753446] LNet: Removed LNI 192.168.203.128@tcp [ 1783.096685] Key type .llcrypt unregistered [ 1783.097798] Key type ._llcrypt unregistered [ 1790.655519] Key type ._llcrypt registered [ 1790.656946] Key type .llcrypt registered [ 1790.706390] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1791.147751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1791.233798] alg: No test for adler32 (adler32-zlib) [ 1792.108154] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1792.204653] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1793.784137] Key type lgssc registered [ 1794.167969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1796.846089] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1796.855167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1812.320339] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1817.440287] LustreError: 108497:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 1844.064229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.971463] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1848.344536] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1850.609042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.699456] LustreError: MGC192.168.203.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1850.703880] LustreError: 109518:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1850.707078] LustreError: 109518:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1850.710368] LustreError: 109518:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1850.740748] Lustre: server umount lustre-MDT0000 complete [ 1850.742095] LustreError: 109518:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1863.680067] Lustre: server umount lustre-OST0000 complete [ 1866.557383] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 1869.673759] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1881.327181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.404482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1881.414487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1881.447979] Lustre: lustre-MDT0000: new disk, initializing [ 1881.468326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.475039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1882.798548] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1886.865859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1886.913228] Lustre: 112302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1886.944112] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1886.947566] Lustre: Skipped 1 previous similar message [ 1886.986331] Lustre: lustre-MDT0001: new disk, initializing [ 1887.019969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1887.031781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1887.035994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1888.361543] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1892.029188] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1893.777755] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1895.966786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1896.077869] Lustre: lustre-OST0000: new disk, initializing [ 1896.079377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1896.081712] Lustre: 114272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1896.111355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.991087] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1901.551608] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1902.942683] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1905.145740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1905.148892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1905.181466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1906.076561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1907.400774] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1907.473464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1910.242524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1910.243529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.244791] LustreError: Skipped 1 previous similar message [ 1910.245635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1910.249383] Lustre: Skipped 1 previous similar message [ 1914.186510] Lustre: server umount lustre-OST0000 complete [ 1917.920933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1917.923918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.928589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.930468] Lustre: Skipped 1 previous similar message [ 1920.481712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1920.482476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.568511] Lustre: server umount lustre-MDT0000 complete [ 1922.872070] LustreError: 113100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414803 with bad export cookie 3340816711244918389 [ 1922.874657] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1922.875729] LustreError: 113100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1922.991273] Lustre: server umount lustre-MDT0001 complete [ 1925.758980] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:46:45 (1784414805) [ 1926.849821] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1932.051428] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 1934.946374] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1946.395613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.484262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1946.493659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1946.534740] Lustre: lustre-MDT0000: new disk, initializing [ 1946.559582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.568401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1948.026873] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1952.190673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.237611] Lustre: 119181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1952.270557] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1952.272339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1952.272772] Lustre: Skipped 1 previous similar message [ 1952.274657] Lustre: Skipped 1 previous similar message [ 1952.308299] Lustre: lustre-MDT0001: new disk, initializing [ 1952.356914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1952.362735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1953.693632] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1957.518857] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1959.519447] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1961.901281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1962.018586] Lustre: lustre-OST0000: new disk, initializing [ 1962.020365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1962.023172] Lustre: 121151:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1962.055762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.058099] Lustre: Skipped 1 previous similar message [ 1963.987722] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1967.099670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1967.104780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1967.138248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1967.830336] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1969.268910] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1969.350225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1970.728907] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1970.803169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1972.194095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.196639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.197454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.200302] Lustre: Skipped 1 previous similar message [ 1973.520983] Lustre: server umount lustre-OST0000 complete [ 1977.314028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.318668] Lustre: Skipped 1 previous similar message [ 1982.433198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.435621] Lustre: Skipped 4 previous similar messages [ 1989.088099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1989.153323] Lustre: server umount lustre-MDT0000 complete [ 1990.373214] LustreError: 119173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414870 with bad export cookie 3340816711244919257 [ 1990.375243] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1990.379890] LustreError: 119173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1990.498563] Lustre: server umount lustre-MDT0001 complete [ 1993.324385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.495856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.827737] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1997.655326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.157602] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2001.576404] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2003.344320] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2005.511909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2007.569771] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2009.881674] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2022.954814] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2028.515036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2028.516700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2028.517787] LustreError: Skipped 1 previous similar message [ 2028.525068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2028.527619] Lustre: Skipped 3 previous similar messages [ 2029.964259] Lustre: server umount lustre-OST0000 complete [ 2033.634174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2037.278898] Lustre: server umount lustre-MDT0000 complete [ 2038.463133] LustreError: 123348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414918 with bad export cookie 3340816711244920139 [ 2038.465911] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2038.467446] LustreError: 123348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2038.582254] Lustre: server umount lustre-MDT0001 complete [ 2042.177204] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2043.237601] Key type lgssc unregistered [ 2043.370327] LNet: 127014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.373576] LNetError: 127014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.379379] LNet: Removed LNI 192.168.203.128@tcp [ 2043.708930] Key type .llcrypt unregistered [ 2043.710819] Key type ._llcrypt unregistered [ 2049.609552] Key type ._llcrypt registered [ 2049.611355] Key type .llcrypt registered [ 2049.648803] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 2055.747094] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2056.028704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.068570] alg: No test for adler32 (adler32-zlib) [ 2056.941154] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2057.042954] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2058.624253] Key type lgssc registered [ 2058.980581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2069.735647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2069.740566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.816448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2070.826607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2070.867862] Lustre: lustre-MDT0000: new disk, initializing [ 2070.887794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.894078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2072.108408] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2076.016178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.052853] Lustre: 130153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2076.073878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2076.075624] Lustre: Skipped 1 previous similar message [ 2076.108565] Lustre: lustre-MDT0001: new disk, initializing [ 2076.138049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2076.152467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2076.157860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2077.411918] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2080.984512] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2082.766296] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2084.951798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2085.049309] Lustre: lustre-OST0000: new disk, initializing [ 2085.051033] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2085.053846] Lustre: 132118:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2085.081509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.907462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2086.910741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2086.949530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2086.958877] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2090.613591] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2091.967606] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2092.036778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2093.494102] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2093.575617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2095.014342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.016851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2095.021260] Lustre: Skipped 1 previous similar message [ 2096.268749] Lustre: server umount lustre-OST0000 complete [ 2100.195068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.196191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.200026] Lustre: Skipped 1 previous similar message [ 2100.202296] Lustre: Skipped 1 previous similar message [ 2103.582334] Lustre: server umount lustre-MDT0000 complete [ 2104.788274] LustreError: 130145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414985 with bad export cookie 5493802153526293682 [ 2104.790780] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.793530] LustreError: 130145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2104.904540] Lustre: server umount lustre-MDT0001 complete [ 2107.433370] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:49:47 (1784414987) [ 2109.461416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.633214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.904643] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2113.604174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.721139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2115.047236] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2117.382327] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2119.129586] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2123.873670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.874422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.877368] Lustre: Skipped 1 previous similar message [ 2125.908628] Lustre: server umount lustre-MDT0000 complete [ 2127.128307] LustreError: 134552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415007 with bad export cookie 5493802153526294550 [ 2127.131097] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2127.133647] LustreError: 134552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2127.257094] Lustre: server umount lustre-MDT0001 complete [ 2129.980932] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:50:09 (1784415009) [ 2132.080033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2147.488462] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2152.608423] LustreError: 136937:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 2178.208402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2179.985987] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2182.127415] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2182.779497] Lustre: server umount lustre-OST0000 complete [ 2185.260514] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:51:05 (1784415065) [ 2187.297631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.436234] LustreError: 138379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.446347] LustreError: 138379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2188.711597] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2191.240668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.585146] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2194.798778] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2196.479819] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2198.580291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2198.693359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2198.695429] Lustre: Skipped 2 previous similar messages [ 2200.410651] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2202.722930] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2216.928773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2216.931330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.935424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2216.937138] Lustre: Skipped 1 previous similar message [ 2221.009740] Lustre: server umount lustre-MDT0000 complete [ 2222.160317] LustreError: 138360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415102 with bad export cookie 5493802153526295201 [ 2222.164108] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.165681] LustreError: 138360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2222.265093] Lustre: server umount lustre-MDT0001 complete [ 2233.683986] Lustre: server umount lustre-OST0000 complete [ 2236.264541] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:51:56 (1784415116) [ 2238.322224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.467069] LustreError: 142052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.488394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.711871] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2242.423953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.813826] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2246.007282] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2247.629733] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2249.727966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2251.566292] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2253.756820] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2255.116320] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2263.142652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2263.343754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2264.655938] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2264.726495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2267.616963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2267.619264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2267.622363] Lustre: Skipped 2 previous similar messages [ 2267.624195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2267.626300] Lustre: Skipped 3 previous similar messages [ 2271.436129] Lustre: server umount lustre-OST0000 complete [ 2273.248890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2278.752814] Lustre: server umount lustre-MDT0000 complete [ 2278.881443] LustreError: 142053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.886902] LustreError: 142053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2279.990428] LustreError: 142032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415160 with bad export cookie 5493802153526296258 [ 2279.993828] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.995252] LustreError: 142032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2280.100511] Lustre: server umount lustre-MDT0001 complete [ 2282.718292] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:52:42 (1784415162) [ 2284.850230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2300.256460] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2305.376355] LustreError: 146263:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 2330.976238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.978470] Lustre: Skipped 2 previous similar messages [ 2332.862238] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2335.223762] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2337.702480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.879097] LustreError: 146288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.947722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2339.257623] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2342.042164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2343.541590] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2345.811193] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2347.640073] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2349.110051] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2349.177604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2350.628462] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2350.693332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2351.586948] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.590241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.594653] Lustre: Skipped 4 previous similar messages [ 2351.600945] Lustre: Skipped 6 previous similar messages [ 2357.544130] Lustre: server umount lustre-MDT0000 complete [ 2358.875285] LustreError: 146269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415239 with bad export cookie 5493802153526297266 [ 2358.877962] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2358.879453] LustreError: 146269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2358.975813] Lustre: server umount lustre-MDT0001 complete [ 2370.403422] Lustre: server umount lustre-OST0000 complete [ 2373.081895] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:54:12 (1784415252) [ 2375.257962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2390.688675] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2395.808422] LustreError: 150452:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 2423.509844] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2426.752236] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2429.548862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.699278] LustreError: 150477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.742056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2431.141855] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2434.556373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.196645] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2438.773503] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2440.800594] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2442.964571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2443.036164] Lustre: lustre-OST0001: new disk, initializing [ 2443.037735] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2443.039982] Lustre: 153322:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2444.822343] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2448.242350] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2449.488237] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2450.933489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2450.937153] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2450.969016] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2451.585976] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2452.848593] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2452.914555] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2455.521749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2455.521910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.526078] LustreError: Skipped 1 previous similar message [ 2455.531739] Lustre: Skipped 1 previous similar message [ 2455.532266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2455.539669] Lustre: Skipped 4 previous similar messages [ 2459.592683] Lustre: server umount lustre-OST0000 complete [ 2461.154918] LustreError: 150469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.156327] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2461.165249] LustreError: 150469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2461.170597] LustreError: Skipped 1 previous similar message [ 2467.221280] Lustre: server umount lustre-OST0001 complete [ 2470.881378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2474.652333] Lustre: server umount lustre-MDT0000 complete [ 2475.896430] LustreError: 151516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415356 with bad export cookie 5493802153526298218 [ 2475.900355] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2475.901991] LustreError: 151516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2476.024853] Lustre: server umount lustre-MDT0001 complete [ 2483.888544] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:56:03 (1784415363) [ 2484.411473] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2484.966286] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2485.479234] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:56:05 (1784415365) [ 2487.577761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.671417] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2487.683978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2487.758974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.761887] Lustre: Skipped 6 previous similar messages [ 2489.195482] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2492.238834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.270741] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2492.280631] Lustre: 156894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2493.667925] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2495.913246] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2497.634724] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2499.738908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2499.805983] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2501.809243] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2502.684995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2504.234206] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2505.586432] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2505.654941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2506.950582] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2507.016188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2507.746385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.747038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2507.748788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.748796] Lustre: Skipped 9 previous similar messages [ 2507.750137] Lustre: Skipped 6 previous similar messages [ 2507.752235] LustreError: Skipped 1 previous similar message [ 2509.716370] Lustre: server umount lustre-OST0000 complete [ 2523.390871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2525.353184] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2527.792816] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2528.615216] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.619103] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2528.619170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.435482] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2529.514928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2530.939593] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2531.021750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2532.821397] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ac104544000.ost_server_uuid 50 [ 2533.366114] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8ac104544000.ost_server_uuid in FULL state after 0 sec [ 2533.418408] LustreError: 156900:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.28@tcp) returned error from blocking AST (req@00000000d44248d9 x1871094677162112 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000014cf6666/0x4c3de7f383360fb6 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.28@tcp remote: 0x4fe350928a8064bc expref: 5 pid: 160151 timeout: 2633 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2533.442891] LustreError: lustre-MDT0000: A client on nid 192.168.203.28@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2533.449655] LustreError: 156889:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.28@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000014cf6666/0x4c3de7f383360fb6 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.28@tcp remote: 0x4fe350928a8064bc expref: 6 pid: 160151 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2540.555296] Lustre: server umount lustre-OST0000 complete [ 2543.585690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2543.593691] LustreError: Skipped 2 previous similar messages [ 2547.935301] Lustre: server umount lustre-MDT0000 complete [ 2549.130640] LustreError: 156885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415429 with bad export cookie 5493802153526299401 [ 2549.133545] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2549.135442] LustreError: 156885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2555.313136] Lustre: server umount lustre-MDT0001 complete [ 2558.957907] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2559.964360] Key type lgssc unregistered [ 2560.085418] LNet: 162994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2560.088101] LNetError: 162994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2560.094281] LNet: Removed LNI 192.168.203.128@tcp [ 2560.402782] Key type .llcrypt unregistered [ 2560.404726] Key type ._llcrypt unregistered [ 2569.138665] Key type ._llcrypt registered [ 2569.139920] Key type .llcrypt registered [ 2569.178401] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2569.458879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2569.485533] alg: No test for adler32 (adler32-zlib) [ 2570.330152] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2570.413644] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2571.992156] Key type lgssc registered [ 2572.381720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2575.090761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2575.096993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.233523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.514833] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2580.313462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.429846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2581.711577] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2584.144976] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2585.919776] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2588.180265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2588.293610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.116552] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2592.394495] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2601.640094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2605.347658] Lustre: Failing over lustre-MDT0000 [ 2605.527225] Lustre: server umount lustre-MDT0000 complete [ 2606.050339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2606.053881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2606.063750] LustreError: 165818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.070405] LustreError: 165818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2607.073360] LustreError: 164385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.074553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.078050] LustreError: 164385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2607.084019] Lustre: Skipped 1 previous similar message [ 2612.193971] LustreError: 165818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.206351] LustreError: 165818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2616.534431] LustreError: 164367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.28@tcp arrived at 1784415496 with bad export cookie 6564987418543661293 [ 2616.544260] LustreError: 164367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2617.314838] LustreError: 164385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.325850] LustreError: 164385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2619.215157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2619.253475] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2619.345822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2619.378189] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2620.716184] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2621.658371] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2621.661264] Lustre: lustre-MDT0000: Denying connection for new client ca309ca2-b55d-4d0a-8950-a77437355996 (at 192.168.203.28@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2624.486280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2629.602313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.604074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2629.605726] Lustre: Skipped 1 previous similar message [ 2634.443662] Lustre: server umount lustre-OST0000 complete [ 2635.588905] LustreError: 168132:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2635.591572] Lustre: 167483:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2635.594087] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2635.599630] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2635.603558] LustreError: 167483:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d9d1f5f9 x1871095214936320/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2635.609375] LustreError: 167483:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2635.612982] LustreError: 167483:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2635.616199] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2635.714350] Lustre: server umount lustre-MDT0000 complete [ 2636.863421] LustreError: 164365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415517 with bad export cookie 6564987418543661650 [ 2636.866742] LustreError: 164365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2636.866966] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2636.873125] LustreError: 165818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.048336] Lustre: server umount lustre-MDT0001 complete [ 2646.376357] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2647.294616] Key type lgssc unregistered [ 2647.415307] LNet: 168885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2647.417555] LNetError: 168885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2647.426284] LNet: Removed LNI 192.168.203.128@tcp [ 2647.712753] Key type .llcrypt unregistered [ 2647.713874] Key type ._llcrypt unregistered [ 2655.740625] Key type ._llcrypt registered [ 2655.741787] Key type .llcrypt registered [ 2655.775377] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2656.091915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2656.149293] alg: No test for adler32 (adler32-zlib) [ 2656.993491] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2657.073895] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2658.648124] Key type lgssc registered [ 2658.960895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2661.311572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2661.317282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.451786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.659441] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2666.215235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.325268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2667.505741] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2669.712875] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2671.457167] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2673.474943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2673.580909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2675.315125] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2677.451490] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2686.892411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2692.070454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.075623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2692.080092] Lustre: Skipped 1 previous similar message [ 2692.080504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2692.084390] Lustre: Skipped 1 previous similar message [ 2695.565893] Lustre: server umount lustre-OST0000 complete [ 2697.185916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.186499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2697.186742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2697.190881] Lustre: Skipped 1 previous similar message [ 2697.193486] LustreError: Skipped 1 previous similar message [ 2702.306372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2702.308617] Lustre: Skipped 2 previous similar messages [ 2702.935678] Lustre: server umount lustre-MDT0000 complete [ 2704.082187] LustreError: 170253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415584 with bad export cookie 8804442908390319698 [ 2704.085674] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.086042] LustreError: 170253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2704.199395] Lustre: server umount lustre-MDT0001 complete [ 2707.535847] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2708.490377] Key type lgssc unregistered [ 2708.612410] LNet: 173820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2708.615334] LNetError: 173820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2708.625240] LNet: Removed LNI 192.168.203.128@tcp [ 2708.929195] Key type .llcrypt unregistered [ 2708.931057] Key type ._llcrypt unregistered [ 2721.197132] Key type ._llcrypt registered [ 2721.198747] Key type .llcrypt registered [ 2721.234067] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2721.642366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2721.649080] alg: No test for adler32 (adler32-zlib) [ 2722.502627] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2722.586631] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2724.160121] Key type lgssc registered [ 2724.484083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2726.971421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2726.977079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.125276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2729.387255] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2732.031684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.144619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2733.393446] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2735.561559] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2737.182833] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2739.248242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2739.351822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2741.054698] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2743.205602] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2753.703234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2763.233725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2763.238973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.248325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2764.260737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2764.261134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2764.263538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.219317] Lustre: server umount lustre-OST0000 complete [ 2767.840831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2767.843248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.847509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2767.849464] Lustre: Skipped 1 previous similar message [ 2773.527566] Lustre: server umount lustre-MDT0000 complete [ 2774.496960] LustreError: 175495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.504056] LustreError: 175495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2774.662172] LustreError: 175476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415655 with bad export cookie 8423623511453431618 [ 2774.664910] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2774.666204] LustreError: 175476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2774.778892] Lustre: server umount lustre-MDT0001 complete [ 2778.012814] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2778.909698] Key type lgssc unregistered [ 2779.031381] LNet: 179046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2779.033918] LNetError: 179046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2779.045271] LNet: Removed LNI 192.168.203.128@tcp [ 2779.310674] Key type .llcrypt unregistered [ 2779.311912] Key type ._llcrypt unregistered [ 2788.466423] Key type ._llcrypt registered [ 2788.467572] Key type .llcrypt registered [ 2788.514615] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2788.912313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2788.975945] alg: No test for adler32 (adler32-zlib) [ 2789.853473] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2789.967248] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2791.552165] Key type lgssc registered [ 2791.973444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2796.104449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2796.110128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.242062] Lustre: *** cfs_fail_loc=135, val=0*** [ 2797.287974] LustreError: 180558:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2797.291276] LustreError: 180558:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -2 [ 2797.295212] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2797.295212] [ 2797.298977] LustreError: MGC192.168.203.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2797.304914] LustreError: 180519:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2797.308088] LustreError: 180519:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2797.311336] LustreError: 180519:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2797.313887] LustreError: 180519:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2797.374145] Lustre: server umount lustre-MDT0000 complete [ 2797.377426] LustreError: 180519:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2799.972086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.157574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.764146] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2805.031299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.163376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2806.680489] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2809.192931] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2811.242525] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2815.075414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.076616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.079248] Lustre: Skipped 1 previous similar message [ 2818.160027] Lustre: server umount lustre-MDT0000 complete [ 2819.689750] LustreError: 180912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415700 with bad export cookie 7354234619166017630 [ 2819.693185] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2819.695328] LustreError: 180912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2819.821976] Lustre: server umount lustre-MDT0001 complete [ 2823.284689] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2824.296729] Key type lgssc unregistered [ 2824.414375] LNet: 183248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2824.417538] LNetError: 183248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2824.430292] LNet: Removed LNI 192.168.203.128@tcp [ 2824.694887] Key type .llcrypt unregistered [ 2824.696618] Key type ._llcrypt unregistered [ 2831.477366] Key type ._llcrypt registered [ 2831.478371] Key type .llcrypt registered [ 2831.514587] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2831.611218] Key type .llcrypt unregistered [ 2831.612385] Key type ._llcrypt unregistered [ 2838.132329] Key type ._llcrypt registered [ 2838.133346] Key type .llcrypt registered [ 2838.168574] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2838.513064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2838.558858] alg: No test for adler32 (adler32-zlib) [ 2839.409280] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2839.499391] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2841.080147] Key type lgssc registered [ 2841.408468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2843.896728] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2843.902143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2859.360325] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2864.480318] LustreError: 185355:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 2891.104228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.969228] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2895.473428] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2897.691203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2897.875596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.903439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2899.185918] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2901.968591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.100752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2903.431861] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2905.845376] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2907.670691] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2908.635061] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2916.838056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.848252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2916.852239] Lustre: Skipped 1 previous similar message [ 2919.653742] Lustre: server umount lustre-OST0000 complete [ 2921.953705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.954549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.958380] Lustre: Skipped 2 previous similar messages [ 2921.961852] Lustre: Skipped 1 previous similar message [ 2927.073101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2927.075859] Lustre: Skipped 1 previous similar message [ 2932.193458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2932.196753] Lustre: Skipped 1 previous similar message [ 2935.264112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2935.317904] Lustre: server umount lustre-MDT0000 complete [ 2936.513279] LustreError: 188018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415816 with bad export cookie 8507917907151304835 [ 2936.516670] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.517317] LustreError: 188018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2936.632971] Lustre: server umount lustre-MDT0001 complete [ 2940.296314] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2941.352201] Key type lgssc unregistered [ 2941.473402] LNet: 189272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2941.476762] LNetError: 189272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2941.484354] LNet: Removed LNI 192.168.203.128@tcp [ 2941.791154] Key type .llcrypt unregistered [ 2941.792269] Key type ._llcrypt unregistered [ 2950.454664] Key type ._llcrypt registered [ 2950.455719] Key type .llcrypt registered [ 2950.489168] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2950.808397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2950.846069] alg: No test for adler32 (adler32-zlib) [ 2951.693657] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2951.788067] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2953.368135] Key type lgssc registered [ 2953.683488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2956.149505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2956.155227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.293433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2958.586916] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2961.157638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2961.277290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2962.553167] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2964.858723] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2966.557685] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2968.588677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2968.695767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2970.456399] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2972.651403] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2981.987672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2984.124565] Lustre: Failing over lustre-MDT0000 [ 2984.220177] Lustre: server umount lustre-MDT0000 complete [ 2986.976774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2986.979089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2986.983023] LustreError: 193149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2986.987423] LustreError: 193149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2987.489278] LustreError: 190678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.490349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2987.493569] LustreError: 190678:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2987.496709] Lustre: Skipped 1 previous similar message [ 2992.609080] LustreError: 190659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.614675] LustreError: 190659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2996.922563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2996.957778] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2997.042563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.069605] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2998.257161] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2998.414928] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2999.654833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3002.344344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3002.350957] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3002.366275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3004.059885] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3010.018197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3010.020391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.024318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3012.064721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3012.066844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3012.070403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3015.307521] Lustre: server umount lustre-OST0000 complete [ 3017.697596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3017.698199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.704193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3017.705862] Lustre: Skipped 2 previous similar messages [ 3022.543392] Lustre: server umount lustre-MDT0000 complete [ 3022.816941] LustreError: 190659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.821261] LustreError: 190659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3023.638988] LustreError: 190642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415904 with bad export cookie 6395089169839551909 [ 3023.641970] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3023.642122] LustreError: 190642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3023.748887] Lustre: server umount lustre-MDT0001 complete [ 3026.988516] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3027.933804] Key type lgssc unregistered [ 3028.062361] LNet: 195556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3028.064552] LNetError: 195556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3028.075407] LNet: Removed LNI 192.168.203.128@tcp [ 3028.359214] Key type .llcrypt unregistered [ 3028.360314] Key type ._llcrypt unregistered [ 3037.518115] Key type ._llcrypt registered [ 3037.519298] Key type .llcrypt registered [ 3037.560139] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3037.899351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3037.942516] alg: No test for adler32 (adler32-zlib) [ 3038.804824] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3038.891245] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3040.472149] Key type lgssc registered [ 3040.835053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3043.276726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3043.282497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.417056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3045.730545] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3048.365843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.474792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3049.736514] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3052.018538] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3053.690203] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3055.849411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3055.989141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3057.899774] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3060.147413] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3069.287796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3071.237634] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3080.824287] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3090.466061] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3100.130281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3100.131668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.133274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.135933] Lustre: Skipped 1 previous similar message [ 3100.138649] Lustre: Skipped 1 previous similar message [ 3105.250065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3105.251934] Lustre: Skipped 1 previous similar message [ 3105.808235] Lustre: server umount lustre-OST0000 complete [ 3109.857020] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3109.859209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3109.863447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3113.180434] Lustre: server umount lustre-MDT0000 complete [ 3114.448068] LustreError: 196924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415994 with bad export cookie 13821404934317088590 [ 3114.451808] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3114.453133] LustreError: 196924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3114.561950] Lustre: server umount lustre-MDT0001 complete [ 3118.130291] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3119.134652] Key type lgssc unregistered [ 3119.266530] LNet: 200812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3119.271473] LNetError: 200812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3119.282391] LNet: Removed LNI 192.168.203.128@tcp [ 3119.666156] Key type .llcrypt unregistered [ 3119.667982] Key type ._llcrypt unregistered [ 3126.977585] Key type ._llcrypt registered [ 3126.979603] Key type .llcrypt registered [ 3127.027936] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3127.360958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3127.405491] alg: No test for adler32 (adler32-zlib) [ 3128.266129] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3128.351158] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3129.936177] Key type lgssc registered [ 3130.291221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3132.806234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3132.811725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3133.951925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3135.198711] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3137.870334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3137.990789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3139.274732] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3141.562368] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3143.271869] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3145.465392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3145.587849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.539826] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3149.860930] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3158.887119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3161.638476] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3170.485301] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3178.329468] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3186.066601] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3199.972460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3199.973795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.976419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3199.991333] Lustre: Skipped 1 previous similar message [ 3202.520616] Lustre: server umount lustre-OST0000 complete [ 3204.576954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3204.578933] LustreError: Skipped 1 previous similar message [ 3204.580023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.583527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.585261] Lustre: Skipped 1 previous similar message [ 3210.210183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3210.214435] Lustre: Skipped 3 previous similar messages [ 3215.329816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3215.332284] Lustre: Skipped 1 previous similar message [ 3217.888126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3217.967718] Lustre: server umount lustre-MDT0000 complete [ 3219.252286] LustreError: 202081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416099 with bad export cookie 6991623652889122147 [ 3219.256891] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3219.260888] LustreError: 202081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3219.384280] Lustre: server umount lustre-MDT0001 complete [ 3223.300802] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3224.438819] Key type lgssc unregistered [ 3224.593596] LNet: 207623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3224.597772] LNetError: 207623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3224.607616] LNet: Removed LNI 192.168.203.128@tcp [ 3225.005876] Key type .llcrypt unregistered [ 3225.008136] Key type ._llcrypt unregistered [ 3234.984342] Key type ._llcrypt registered [ 3234.985444] Key type .llcrypt registered [ 3235.023382] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3235.448381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3235.456475] alg: No test for adler32 (adler32-zlib) [ 3236.324188] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3236.426517] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3238.024241] Key type lgssc registered [ 3238.434713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3241.091573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3241.100691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3242.265682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.818069] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3247.381796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.504420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3248.933413] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3251.441642] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3253.289913] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3256.071564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3256.232528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3258.788431] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3261.795332] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3268.522673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3277.793451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3277.799051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3277.808037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3278.817628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.818785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3278.819044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3281.831162] Lustre: server umount lustre-OST0000 complete [ 3283.938178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.951920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.955918] Lustre: Skipped 1 previous similar message [ 3288.545594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3288.550767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.561839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.654771] Lustre: server umount lustre-MDT0000 complete [ 3291.517811] LustreError: 208993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416171 with bad export cookie 5980590240690630213 [ 3291.524110] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3291.526334] LustreError: 208993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3291.658160] Lustre: server umount lustre-MDT0001 complete [ 3296.175000] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3297.551875] Key type lgssc unregistered [ 3297.684394] LNet: 212712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3297.688786] LNetError: 212712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3297.697489] LNet: Removed LNI 192.168.203.128@tcp [ 3298.089297] Key type .llcrypt unregistered [ 3298.090931] Key type ._llcrypt unregistered [ 3305.742411] Key type ._llcrypt registered [ 3305.744157] Key type .llcrypt registered [ 3305.782730] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3306.187215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3306.195951] alg: No test for adler32 (adler32-zlib) [ 3307.036775] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3307.114182] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3308.696104] Key type lgssc registered [ 3309.026589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3311.723562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3311.731443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.892033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.172485] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3316.829980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3316.961095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3318.257977] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3320.499071] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3322.240288] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3324.309922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3324.418132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.195723] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3328.429532] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3337.766731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3347.938556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3347.938960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3347.942213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3347.947251] Lustre: Skipped 1 previous similar message [ 3347.951566] Lustre: Skipped 1 previous similar message [ 3353.061168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3353.063926] Lustre: Skipped 1 previous similar message [ 3357.152126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3357.233320] Lustre: server umount lustre-OST0000 complete [ 3363.298052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3363.298619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.299553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3363.301221] LustreError: Skipped 1 previous similar message [ 3363.308719] Lustre: Skipped 1 previous similar message [ 3365.039590] Lustre: server umount lustre-MDT0000 complete [ 3366.671629] LustreError: 213935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416247 with bad export cookie 4149259109534319141 [ 3366.677422] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3366.677780] LustreError: 213935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3366.809603] Lustre: server umount lustre-MDT0001 complete [ 3370.970401] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3372.181044] Key type lgssc unregistered [ 3372.298306] LNet: 217650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3372.300588] LNetError: 217650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3372.309297] LNet: Removed LNI 192.168.203.128@tcp [ 3372.578198] Key type .llcrypt unregistered [ 3372.579394] Key type ._llcrypt unregistered [ 3380.562119] Key type ._llcrypt registered [ 3380.563145] Key type .llcrypt registered [ 3380.594901] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3380.883678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3380.939789] alg: No test for adler32 (adler32-zlib) [ 3381.852216] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3381.970807] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3383.568195] Key type lgssc registered [ 3384.116681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3387.149864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3387.157139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.330577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3389.998849] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3393.295722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3393.449165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3395.045635] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3398.209650] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3400.622800] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3403.549102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3403.700007] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.947792] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3409.098615] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3412.910530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3418.261936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3418.284084] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3418.353610] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3420.204797] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3422.979886] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3424.756878] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3435.128490] Lustre: Permanently deactivating lustre-OST0001 [ 3435.135745] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3441.150426] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3446.039779] Lustre: Permanently reactivating lustre-OST0001 [ 3446.044778] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3446.047070] Lustre: Skipped 2 previous similar messages [ 3447.941794] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3447.949757] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3447.954844] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3447.962391] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3447.979810] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3458.529854] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3458.529880] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.533795] LustreError: Skipped 1 previous similar message [ 3458.534441] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3458.542032] Lustre: Skipped 2 previous similar messages [ 3459.558070] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3459.561241] Lustre: Skipped 2 previous similar messages [ 3461.449642] Lustre: server umount lustre-OST0001 complete [ 3464.161108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3464.161158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3464.163333] LustreError: Skipped 1 previous similar message [ 3464.167301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3464.170822] Lustre: Skipped 1 previous similar message [ 3464.674778] LustreError: 220712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3464.682410] LustreError: 220712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3468.748768] Lustre: server umount lustre-OST0000 complete [ 3470.304693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3470.307036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3470.310664] Lustre: Skipped 1 previous similar message [ 3470.312391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3474.913120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3474.913710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3476.051732] Lustre: server umount lustre-MDT0000 complete [ 3477.203961] LustreError: 218875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416357 with bad export cookie 5809044817073573491 [ 3477.207168] LustreError: 218875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3477.207808] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3477.323944] Lustre: server umount lustre-MDT0001 complete [ 3480.642074] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3481.627136] Key type lgssc unregistered [ 3481.756357] LNet: 223912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3481.758886] LNetError: 223912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3481.765413] LNet: Removed LNI 192.168.203.128@tcp [ 3482.080944] Key type .llcrypt unregistered [ 3482.082259] Key type ._llcrypt unregistered [ 3495.542211] Key type ._llcrypt registered [ 3495.543324] Key type .llcrypt registered [ 3495.580505] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3495.968841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3496.044566] alg: No test for adler32 (adler32-zlib) [ 3496.912557] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3497.001366] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3498.592179] Key type lgssc registered [ 3498.977823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3501.465035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3501.470817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.556742] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3502.563415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3502.614235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.922265] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3506.532757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.563076] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3506.573218] Lustre: 226105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3506.661754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3507.890154] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3510.052927] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3511.737848] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3513.823771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3513.885649] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3513.948867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.806336] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3518.035053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3518.109572] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3519.673345] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3526.316517] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3531.893618] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3539.516431] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3555.644882] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3555.647616] Lustre: Skipped 1 previous similar message [ 3577.467963] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3577.473297] Lustre: Skipped 2 previous similar messages [ 3613.315252] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3613.317950] Lustre: Skipped 3 previous similar messages [ 3678.134758] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3678.138537] Lustre: Skipped 8 previous similar messages [ 3681.761507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3681.761541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3681.764278] LustreError: Skipped 1 previous similar message [ 3681.774843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3681.782695] Lustre: Skipped 1 previous similar message [ 3684.836712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3684.840186] Lustre: Skipped 1 previous similar message [ 3686.000665] Lustre: server umount lustre-OST0000 complete [ 3689.954784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.955793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3689.961392] Lustre: Skipped 2 previous similar messages [ 3689.966669] Lustre: Skipped 1 previous similar message [ 3693.873718] Lustre: server umount lustre-MDT0000 complete [ 3695.586352] LustreError: 226096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416575 with bad export cookie 4018827984429048601 [ 3695.590426] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3695.591152] LustreError: 226096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3695.715460] Lustre: server umount lustre-MDT0001 complete [ 3699.223904] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3700.099550] Key type lgssc unregistered [ 3700.271537] LNet: 230886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3700.277505] LNetError: 230886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3700.288612] LNet: Removed LNI 192.168.203.128@tcp [ 3700.720157] Key type .llcrypt unregistered [ 3700.722425] Key type ._llcrypt unregistered [ 3709.950152] Key type ._llcrypt registered [ 3709.951445] Key type .llcrypt registered [ 3709.989943] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3710.282976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3710.290069] alg: No test for adler32 (adler32-zlib) [ 3711.209980] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3711.325771] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3712.920216] Key type lgssc registered [ 3713.471989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3716.244165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3716.252175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.420964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3719.103891] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3722.442916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.573792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3724.089995] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3727.187166] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3729.171740] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3731.780582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3731.919493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3734.359064] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3737.382299] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3742.191518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3752.418789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3752.420508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.422304] LustreError: Skipped 1 previous similar message [ 3752.428489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3752.432109] Lustre: Skipped 1 previous similar message [ 3753.443709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.445935] Lustre: Skipped 1 previous similar message [ 3754.149459] Lustre: server umount lustre-OST0000 complete [ 3758.064319] Lustre: server umount lustre-MDT0000 complete [ 3758.561786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3758.562568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.564232] LustreError: 232298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3758.564242] LustreError: 232298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3758.583673] Lustre: Skipped 2 previous similar messages [ 3759.649793] LustreError: 232274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3759.665324] LustreError: 232274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3764.706284] LustreError: 233003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3764.718184] LustreError: 233003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3769.825212] LustreError: 232273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3769.833179] LustreError: 232273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3773.920113] Lustre: 231588:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784416638/real 1784416638] req@000000001aa0518a x1871096411355392/t0(0) o400->MGC192.168.203.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784416654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3773.926766] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3773.930285] LustreError: 232266:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3773.930660] LustreError: 231585:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ca370918 x1871096411357696/t0(0) o38->lustre-MDT0000-lwp-MDT0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3773.930718] LustreError: 233003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3774.033447] Lustre: server umount lustre-MDT0001 complete [ 3777.815640] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3779.108101] Key type lgssc unregistered [ 3779.241705] LNet: 235960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3779.247403] LNetError: 235960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3779.259533] LNet: Removed LNI 192.168.203.128@tcp [ 3779.742971] Key type .llcrypt unregistered [ 3779.745538] Key type ._llcrypt unregistered [ 3789.707975] Key type ._llcrypt registered [ 3789.709580] Key type .llcrypt registered [ 3789.748482] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3790.164705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3790.184855] alg: No test for adler32 (adler32-zlib) [ 3791.040338] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3791.127087] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3792.704119] Key type lgssc registered [ 3793.135917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4059.548830] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 4061.008719] Key type lgssc unregistered [ 4061.148515] LNet: 237960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4061.153747] LNetError: 237960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4061.163336] LNet: Removed LNI 192.168.203.128@tcp [ 4061.473675] Key type .llcrypt unregistered [ 4061.474920] Key type ._llcrypt unregistered [ 4071.014871] Key type ._llcrypt registered [ 4071.016042] Key type .llcrypt registered [ 4071.059872] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 4078.497221] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4078.797994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4078.864864] alg: No test for adler32 (adler32-zlib) [ 4079.726947] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 4079.827417] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 4081.416174] Key type lgssc registered [ 4082.030803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4095.584548] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3942 sec ========== 19:22:55 (1784416975) [ 4096.505160] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:22:56 (1784416976) === [ 4098.174536] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:22:57 (1784416977) === [ 4116.540718] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 4118.008468] Key type lgssc unregistered [ 4118.190526] LNet: 243164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4118.195801] LNetError: 243164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4118.208495] LNet: Removed LNI 192.168.203.128@tcp [ 4118.663431] Key type .llcrypt unregistered [ 4118.665939] Key type ._llcrypt unregistered