[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 519741174 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003262] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.007823] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009005] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010016] pid_max: default: 32768 minimum: 301 [ 0.012131] LSM: Security Framework initializing [ 0.013059] Yama: becoming mindful. [ 0.015036] SELinux: Initializing. [ 0.016078] *** VALIDATE selinux *** [ 0.024636] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028873] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030166] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032120] *** VALIDATE tmpfs *** [ 0.034447] *** VALIDATE proc *** [ 0.036025] *** VALIDATE cgroup *** [ 0.037013] *** VALIDATE cgroup2 *** [ 0.038293] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041038] Spectre V2 : User space: Vulnerable [ 0.042014] Speculative Store Bypass: Vulnerable [ 0.044714] debug: unmapping init [mem 0xffffffff95659000-0xffffffff95660fff] [ 0.046186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047696] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050020] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053018] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055308] rcu: Hierarchical SRCU implementation. [ 0.057507] smp: Bringing up secondary CPUs ... [ 0.058645] x86: Booting SMP configuration: [ 0.059029] .... node #0, CPUs: #1 #2 #3 [ 0.062357] smp: Brought up 1 node, 4 CPUs [ 0.064015] smpboot: Max logical packages: 1 [ 0.065033] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.221074] node 0 deferred pages initialised in 153ms [ 0.224182] devtmpfs: initialized [ 0.225356] x86/mm: Memory block size: 128MB [ 0.229465] gcov: version magic: 0x41383552 [ 0.232365] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.236157] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.238509] pinctrl core: initialized pinctrl subsystem [ 0.240274] [ 0.240920] ************************************************************* [ 0.243024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.246024] ** ** [ 0.248021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.251021] ** ** [ 0.254021] ** This means that this kernel is built to expose internal ** [ 0.256021] ** IOMMU data structures, which may compromise security on ** [ 0.259021] ** your system. ** [ 0.262022] ** ** [ 0.264020] ** If you see this message and you are not debugging the ** [ 0.267024] ** kernel, report this immediately to your vendor! ** [ 0.270025] ** ** [ 0.272018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.275022] ************************************************************* [ 0.277881] NET: Registered protocol family 16 [ 0.280621] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.283090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.286095] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.291102] cpuidle: using governor menu [ 0.292606] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.294431] PCI: Using configuration type 1 for base access [ 0.296133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.302155] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.305040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.308123] cryptd: max_cpu_qlen set to 1000 [ 0.311047] ACPI: Added _OSI(Module Device) [ 0.311986] ACPI: Added _OSI(Processor Device) [ 0.313069] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.314008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.317056] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.322061] ACPI: Interpreter enabled [ 0.323049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.324009] ACPI: Using IOAPIC for interrupt routing [ 0.325074] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.327273] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.334989] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.336032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.339020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.342086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.348324] acpiphp: Slot [2] registered [ 0.349080] acpiphp: Slot [5] registered [ 0.350103] acpiphp: Slot [6] registered [ 0.351037] acpiphp: Slot [7] registered [ 0.353141] acpiphp: Slot [8] registered [ 0.355117] acpiphp: Slot [9] registered [ 0.356119] acpiphp: Slot [10] registered [ 0.357108] acpiphp: Slot [3] registered [ 0.359084] acpiphp: Slot [4] registered [ 0.360140] acpiphp: Slot [11] registered [ 0.362112] acpiphp: Slot [12] registered [ 0.364124] acpiphp: Slot [13] registered [ 0.365112] acpiphp: Slot [14] registered [ 0.367114] acpiphp: Slot [15] registered [ 0.369116] acpiphp: Slot [16] registered [ 0.370142] acpiphp: Slot [17] registered [ 0.372182] acpiphp: Slot [18] registered [ 0.374188] acpiphp: Slot [19] registered [ 0.376158] acpiphp: Slot [20] registered [ 0.377199] acpiphp: Slot [21] registered [ 0.379254] acpiphp: Slot [22] registered [ 0.381175] acpiphp: Slot [23] registered [ 0.383165] acpiphp: Slot [24] registered [ 0.385182] acpiphp: Slot [25] registered [ 0.386175] acpiphp: Slot [26] registered [ 0.388187] acpiphp: Slot [27] registered [ 0.390200] acpiphp: Slot [28] registered [ 0.391172] acpiphp: Slot [29] registered [ 0.393171] acpiphp: Slot [30] registered [ 0.395173] acpiphp: Slot [31] registered [ 0.397135] PCI host bridge to bus 0000:00 [ 0.398036] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.400037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.403042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.405043] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.408043] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.411043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.413233] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.416135] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.419491] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.430010] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.436048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.438031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.441032] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.444032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.447435] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.450832] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.453039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.457869] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.462015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.474019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.478874] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.484125] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.492030] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.498028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.522031] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.536432] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.544017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.555018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.573022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.582198] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.603024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.608032] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.622024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.635028] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.646020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.654019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.669018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.684230] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.695029] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.701026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.720027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.730904] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.740019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.747017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.762017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.774345] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.776418] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.779506] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.782479] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.785291] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.790116] iommu: Default domain type: Passthrough [ 0.791394] SCSI subsystem initialized [ 0.792107] ACPI: bus type USB registered [ 0.793068] usbcore: registered new interface driver usbfs [ 0.795088] usbcore: registered new interface driver hub [ 0.797077] usbcore: registered new device driver usb [ 0.798140] pps_core: LinuxPPS API ver. 1 registered [ 0.800010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.802067] PTP clock support registered [ 0.804142] EDAC MC: Ver: 3.0.0 [ 0.805556] PCI: Using ACPI for IRQ routing [ 0.806932] NetLabel: Initializing [ 0.807009] NetLabel: domain hash size = 128 [ 0.808014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.809147] NetLabel: unlabeled traffic allowed by default [ 0.811206] vgaarb: loaded [ 0.813200] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.814010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.819143] clocksource: Switched to clocksource kvm-clock [ 0.930485] VFS: Disk quotas dquot_6.6.0 [ 0.932646] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.935571] *** VALIDATE ramfs *** [ 0.937076] *** VALIDATE hugetlbfs *** [ 0.939489] pnp: PnP ACPI init [ 0.942191] pnp: PnP ACPI: found 6 devices [ 0.960631] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.964050] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.966693] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.969078] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.971678] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.974174] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.977399] NET: Registered protocol family 2 [ 0.980081] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.985125] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.989372] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.995135] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.998533] TCP: Hash tables configured (established 65536 bind 65536) [ 1.000944] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.003246] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.005442] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.007814] NET: Registered protocol family 1 [ 1.010357] RPC: Registered named UNIX socket transport module. [ 1.011898] RPC: Registered udp transport module. [ 1.013065] RPC: Registered tcp transport module. [ 1.014250] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.016415] NET: Registered protocol family 44 [ 1.018237] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.020533] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.022897] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.025279] PCI: CLS 0 bytes, default 64 [ 1.027163] Unpacking initramfs... [ 2.416234] debug: unmapping init [mem 0xffffa0907cc54000-0xffffa0907ffbffff] [ 2.419829] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.422216] software IO TLB: mapped [mem 0x00000000a2a00000-0x00000000a6a00000] (64MB) [ 2.425302] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.875485] Initialise system trusted keyrings [ 2.877170] Key type blacklist registered [ 2.879080] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.885771] zbud: loaded [ 2.888240] *** VALIDATE nfs *** [ 2.889035] *** VALIDATE nfs4 *** [ 2.890013] pstore: using deflate compression [ 2.892600] Platform Keyring initialized [ 2.966785] NET: Registered protocol family 38 [ 2.967903] Key type asymmetric registered [ 2.969089] Asymmetric key parser 'x509' registered [ 2.970361] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.972190] io scheduler mq-deadline registered [ 2.973125] io scheduler kyber registered [ 2.973939] io scheduler bfq registered [ 2.975130] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.976860] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.978621] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.983417] ACPI: Power Button [PWRF] [ 2.987712] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.993301] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.009253] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.016936] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.038378] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.070874] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.101536] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.105937] Non-volatile memory driver v1.3 [ 3.107565] Linux agpgart interface v0.103 [ 3.137262] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.140328] vda: detected capacity change from 0 to 74698752 [ 3.162195] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.166477] vdb: detected capacity change from 0 to 1073741824 [ 3.187246] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.190106] vdc: detected capacity change from 0 to 2621440000 [ 3.228624] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.232509] vdd: detected capacity change from 0 to 2621440000 [ 3.250072] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.254745] vde: detected capacity change from 0 to 4294967296 [ 3.271251] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.274350] vdf: detected capacity change from 0 to 4294967296 [ 3.285135] libphy: Fixed MDIO Bus: probed [ 3.290731] usbcore: registered new interface driver usbserial_generic [ 3.293316] usbserial: USB Serial support registered for generic [ 3.295723] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.300113] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.302040] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.304538] mousedev: PS/2 mouse device common for all mice [ 3.306422] rtc_cmos 00:05: RTC can wake from S4 [ 3.307576] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.310258] rtc_cmos 00:05: registered as rtc0 [ 3.311430] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.312870] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.314147] intel_pstate: CPU model not supported [ 3.318278] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.321808] hid: raw HID events driver (C) Jiri Kosina [ 3.324802] usbcore: registered new interface driver usbhid [ 3.328093] usbhid: USB HID core driver [ 3.330390] drop_monitor: Initializing network drop monitor service [ 3.333752] Initializing XFRM netlink socket [ 3.336540] NET: Registered protocol family 10 [ 3.340111] Segment Routing with IPv6 [ 3.341794] NET: Registered protocol family 17 [ 3.344055] mpls_gso: MPLS GSO support [ 3.351271] RAS: Correctable Errors collector initialized. [ 3.353482] AVX version of gcm_enc/dec engaged. [ 3.354868] AES CTR mode by8 optimization enabled [ 3.419427] sched_clock: Marking stable (3419409234, 0)->(4374018857, -954609623) [ 3.421719] registered taskstats version 1 [ 3.422936] Loading compiled-in X.509 certificates [ 3.425589] zswap: loaded using pool lzo/zbud [ 3.447889] Key type big_key registered [ 3.460592] Key type encrypted registered [ 3.463246] ima: No TPM chip found, activating TPM-bypass! [ 3.466343] ima: Allocated hash algorithm: sha1 [ 3.468843] ima: No architecture policies found [ 3.471255] evm: Initialising EVM extended attributes: [ 3.474036] evm: security.selinux [ 3.475917] evm: security.ima [ 3.477488] evm: security.capability [ 3.479427] evm: HMAC attrs: 0x1 [ 3.483129] rtc_cmos 00:05: setting system clock to 2026-09-03 13:31:53 UTC (1788442313) [ 3.490861] debug: unmapping init [mem 0xffffffff96603000-0xffffffff967fffff] [ 3.495291] debug: unmapping init [mem 0xffffffff95382000-0xffffffff95658fff] [ 3.510066] Write protecting the kernel read-only data: 28672k [ 3.514531] debug: unmapping init [mem 0xffffffff93a03000-0xffffffff93bfffff] [ 3.518555] debug: unmapping init [mem 0xffffffff94314000-0xffffffff943fffff] [ 3.554401] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.566339] systemd[1]: Detected virtualization kvm. [ 3.569069] systemd[1]: Detected architecture x86-64. [ 3.571698] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.599953] systemd[1]: No hostname configured. [ 3.602514] systemd[1]: Set hostname to . [ 3.605586] random: systemd: uninitialized urandom read (16 bytes read) [ 3.609243] systemd[1]: Initializing machine ID from random generator. [ 3.724965] random: systemd: uninitialized urandom read (16 bytes read) [ 3.727924] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.733961] random: systemd: uninitialized urandom read (16 bytes read) [ 3.737252] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.744486] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.230328] device-mapper: uevent: version 1.0.3 [ 4.231873] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.838664] random: fast init done [ 4.842307] virtio_net virtio0 ens2: renamed from eth0 [ 4.949121] scsi host0: ata_piix [ 4.979805] scsi host1: ata_piix [ 4.981298] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.983720] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.901317] dracut-initqueue[575]: RTNETLINK answers: File exists [ 9.997157] random: crng init done [ 9.999356] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.816336] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.592073] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.566076] SELinux: Disabled at runtime. [ 15.725801] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.735662] systemd[1]: Detected virtualization kvm. [ 15.737303] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.685113] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.692164] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.701157] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.710830] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.727213] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.747248] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.767835] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 18.015620] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.722200] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.789076] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.159478] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.413112] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.517945] EDAC sbridge: Ver: 1.1.2 [ 25.569225] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 26.381782] NFS: Registering the id_resolver key type [ 26.385241] Key type id_resolver registered [ 26.387894] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg258-server login: [ 91.013973] libcfs: loading out-of-tree module taints kernel. [ 91.077277] Key type ._llcrypt registered [ 91.078480] Key type .llcrypt registered [ 91.226484] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_hostid [ 111.530608] hrtimer: interrupt took 10779207 ns [ 112.518740] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 114.474687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.488974] alg: No test for adler32 (adler32-zlib) [ 116.112540] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 117.248903] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 119.168460] Key type lgssc registered [ 121.399717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 144.258644] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 190.009880] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 202.806745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 202.848834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.163472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 204.205503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 204.349357] Lustre: lustre-MDT0000: new disk, initializing [ 204.445519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.470867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 209.067670] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.334705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.428261] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 222.464674] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 222.469655] Lustre: Skipped 1 previous similar message [ 222.568218] Lustre: lustre-MDT0001: new disk, initializing [ 222.639796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.665252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 222.676509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 226.916386] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.674262] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 240.849163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.099300] Lustre: lustre-OST0000: new disk, initializing [ 241.103921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 241.108909] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 241.189429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 247.342891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 247.351744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 247.437931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.552216] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.766195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 262.857906] Lustre: lustre-OST0001: new disk, initializing [ 262.861916] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 262.867725] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 262.939905] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 270.003191] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.885194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 271.909469] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 272.003259] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 281.983243] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 287.814704] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 294.676620] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing check_logdir /tmp/testlogs/ [ 299.580599] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing yml_node [ 303.625234] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 306.059576] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 308.688992] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 310.230048] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:36:59 EDT 2026 [ 327.176388] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 328.800978] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 345.569051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 345.574507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.587213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 348.639587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 348.640381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.642628] Lustre: Skipped 1 previous similar message [ 348.693611] Lustre: Skipped 1 previous similar message [ 353.760934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.771921] Lustre: Skipped 5 previous similar messages [ 358.884639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.894661] Lustre: Skipped 3 previous similar messages [ 359.394469] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 359.746545] Lustre: server umount lustre-MDT0000 complete [ 363.516973] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442673 with bad export cookie 8767784845369669088 [ 363.523992] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 363.531424] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 364.003424] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.004062] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 364.017757] Lustre: Skipped 2 previous similar messages [ 369.839289] Lustre: server umount lustre-MDT0001 complete [ 373.359694] Lustre: server umount lustre-OST0000 complete [ 377.103225] Lustre: server umount lustre-OST0001 complete [ 385.381382] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_hostid [ 396.871474] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 434.269172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.486674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 434.547208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 434.649495] Lustre: lustre-MDT0000: new disk, initializing [ 434.744676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.770532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 439.774698] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 450.882614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.994634] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 451.061526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 451.066733] Lustre: Skipped 1 previous similar message [ 451.155942] Lustre: lustre-MDT0001: new disk, initializing [ 451.295480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 451.379283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 451.393232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 456.244543] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 465.388762] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 472.367332] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 478.852075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 479.086673] Lustre: lustre-OST0000: new disk, initializing [ 479.093986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 479.105808] Lustre: 17039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 479.182316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.063139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 481.094925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 481.269467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 485.774230] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 495.022700] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 500.348166] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 500.563316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 505.978218] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 506.249496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 511.973294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.984232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 511.987567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 511.987576] Lustre: Skipped 3 previous similar messages [ 512.000861] Lustre: Skipped 1 previous similar message [ 514.453982] Lustre: server umount lustre-OST0000 complete [ 522.209126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.231716] Lustre: Skipped 1 previous similar message [ 524.846633] Lustre: server umount lustre-MDT0000 complete [ 527.346820] LustreError: 15080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.369631] LustreError: 15080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 528.397135] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442838 with bad export cookie 8767784845369670614 [ 528.409994] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.411979] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 528.967627] Lustre: server umount lustre-MDT0001 complete [ 532.597960] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:40:41 (1788442841) [ 539.883053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.436837] LustreError: 19302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.526957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.993903] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 545.770081] LustreError: 19303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.890926] LustreError: 19302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.832310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.381290] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 567.684251] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 574.119691] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 580.319742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 587.567760] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 595.098751] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 606.688396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 606.701235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.725715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 606.731643] Lustre: Skipped 3 previous similar messages [ 610.793469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 612.603388] Lustre: server umount lustre-OST0000 complete [ 621.034057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 622.684542] Lustre: server umount lustre-MDT0000 complete [ 626.146893] LustreError: 20030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.174403] LustreError: 20030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 626.579366] LustreError: 19283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442936 with bad export cookie 8767784845369671503 [ 626.585053] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 626.592241] LustreError: 19283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 626.901421] Lustre: server umount lustre-MDT0001 complete [ 638.325446] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 640.886587] Key type lgssc unregistered [ 641.209920] LNet: 22838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.225759] LNetError: 22838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.244105] LNet: Removed LNI 192.168.202.158@tcp [ 642.204625] Key type .llcrypt unregistered [ 642.206749] Key type ._llcrypt unregistered [ 665.071478] Key type ._llcrypt registered [ 665.083191] Key type .llcrypt registered [ 665.176199] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 666.302914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 666.439438] alg: No test for adler32 (adler32-zlib) [ 667.576571] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 667.801188] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 669.487289] Key type lgssc registered [ 670.649216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 679.335858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 679.363116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.832458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.141424] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 690.148946] LustreError: 24253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.163963] LustreError: 24253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 692.896929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.349430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 697.535083] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 704.855118] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 710.812441] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 718.440697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 718.645827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 725.295249] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 725.828269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 733.143515] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 750.569543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 750.572951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.630503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 751.584625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 751.594933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 751.598876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 755.401385] Lustre: server umount lustre-OST0000 complete [ 759.794224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 759.807510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 759.829700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 759.834087] Lustre: Skipped 1 previous similar message [ 761.825289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.838310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 761.840895] Lustre: Skipped 1 previous similar message [ 765.071024] Lustre: server umount lustre-MDT0000 complete [ 766.978246] LustreError: 24254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 767.003620] LustreError: 24254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 769.580382] LustreError: 24235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443079 with bad export cookie 7454100183055850058 [ 769.600046] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 770.339692] Lustre: server umount lustre-MDT0001 complete [ 785.794585] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 788.298175] Key type lgssc unregistered [ 788.528572] LNet: 28230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.532768] LNetError: 28230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.542839] LNet: Removed LNI 192.168.202.158@tcp [ 789.359528] Key type .llcrypt unregistered [ 789.362113] Key type ._llcrypt unregistered [ 815.138765] Key type ._llcrypt registered [ 815.143885] Key type .llcrypt registered [ 815.272464] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 816.838976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 816.860835] alg: No test for adler32 (adler32-zlib) [ 818.055104] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 818.390670] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 820.177467] Key type lgssc registered [ 821.423194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 829.125270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 829.167979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.872536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 836.338967] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 840.170276] LustreError: 29645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 840.194495] LustreError: 29645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 845.292458] LustreError: 29644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 845.526324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 846.001344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 851.749199] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 859.979676] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 864.742507] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 881.113767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 881.352528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.622047] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 898.140699] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 901.952950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 911.331491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 911.344424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.363532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 912.359063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.360230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 912.390528] Lustre: Skipped 1 previous similar message [ 915.769782] Lustre: server umount lustre-OST0000 complete [ 922.593664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 922.594370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.610651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 922.624125] Lustre: Skipped 1 previous similar message [ 927.724612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 927.737221] Lustre: Skipped 2 previous similar messages [ 932.833552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 932.837610] Lustre: Skipped 1 previous similar message [ 934.367143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 934.557635] Lustre: server umount lustre-MDT0000 complete [ 937.960247] LustreError: 29641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.982364] LustreError: 29641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 938.416938] LustreError: 29623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443248 with bad export cookie 4736234749533753717 [ 938.435726] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 938.887489] Lustre: server umount lustre-MDT0001 complete [ 952.663846] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 955.906894] Key type lgssc unregistered [ 956.566110] LNet: 33529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.580474] LNetError: 33529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.608950] LNet: Removed LNI 192.168.202.158@tcp [ 957.607453] Key type .llcrypt unregistered [ 957.612662] Key type ._llcrypt unregistered [ 977.457963] Key type ._llcrypt registered [ 977.461209] Key type .llcrypt registered [ 977.662065] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 978.761501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 978.803129] alg: No test for adler32 (adler32-zlib) [ 979.881647] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 980.080290] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 981.743996] Key type lgssc registered [ 982.577751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 991.752746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 991.767755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.396158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.539339] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1004.007986] LustreError: 34923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.025879] LustreError: 34923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1007.011560] LustreError: 34924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.785028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.109178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1013.456472] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1022.165518] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1027.544906] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1034.346948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1034.744132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.853200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1040.955701] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1047.411866] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1058.759428] Lustre: server umount lustre-OST0000 complete [ 1063.396913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.419865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1063.430445] Lustre: Skipped 1 previous similar message [ 1068.526296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.533608] Lustre: Skipped 1 previous similar message [ 1072.715669] Lustre: server umount lustre-MDT0000 complete [ 1073.637956] LustreError: 34924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.676646] LustreError: 34924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1076.845317] LustreError: 34903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443386 with bad export cookie 16480155455286663214 [ 1076.857295] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1086.665980] Lustre: server umount lustre-MDT0001 complete [ 1098.704299] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 1102.420831] Key type lgssc unregistered [ 1102.901160] LNet: 38462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1102.908579] LNetError: 38462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1103.993904] LNet: Removed LNI 192.168.202.158@tcp [ 1105.054170] Key type .llcrypt unregistered [ 1105.057800] Key type ._llcrypt unregistered [ 1133.442693] Key type ._llcrypt registered [ 1133.444953] Key type .llcrypt registered [ 1133.595786] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 1135.316414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1135.374172] alg: No test for adler32 (adler32-zlib) [ 1136.431468] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1136.706387] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 1138.479288] Key type lgssc registered [ 1139.669526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1146.959966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1146.971283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.362642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1153.141599] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1158.624934] LustreError: 39876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.660525] LustreError: 39876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1161.177859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.519280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1165.996430] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1173.750737] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1179.264317] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1186.340694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1186.686786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.849404] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1194.889636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1200.536915] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1211.378976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.383115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1211.396439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1211.408861] Lustre: Skipped 1 previous similar message [ 1212.713247] Lustre: server umount lustre-OST0000 complete [ 1230.454082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.477575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1230.480792] Lustre: Skipped 2 previous similar messages [ 1230.504779] Lustre: Skipped 1 previous similar message [ 1231.531558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1231.542268] Lustre: Skipped 1 previous similar message [ 1236.866777] Lustre: server umount lustre-MDT0000 complete [ 1236.963841] LustreError: 39877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.977818] LustreError: 39877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1240.232493] LustreError: 39858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443550 with bad export cookie 4166626748994026405 [ 1240.236927] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.244913] LustreError: 39858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1246.590972] Lustre: server umount lustre-MDT0001 complete [ 1257.851783] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 1261.104262] Key type lgssc unregistered [ 1261.503950] LNet: 43585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.523775] LNetError: 43585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1261.559792] LNet: Removed LNI 192.168.202.158@tcp [ 1262.695609] Key type .llcrypt unregistered [ 1262.702287] Key type ._llcrypt unregistered [ 1288.158115] Key type ._llcrypt registered [ 1288.159665] Key type .llcrypt registered [ 1288.267137] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 1289.647992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1289.684908] alg: No test for adler32 (adler32-zlib) [ 1290.734647] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1291.042091] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 1292.800291] Key type lgssc registered [ 1294.183475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1302.113852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1302.139838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.565528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.876151] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1313.768137] LustreError: 45002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.778445] LustreError: 45002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1316.815946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.382477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1321.702346] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1331.457823] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1337.026553] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1343.457807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1343.745259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1348.854050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1349.832284] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1356.986909] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1367.525489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.527042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1367.547132] Lustre: Skipped 2 previous similar messages [ 1367.572438] Lustre: Skipped 2 previous similar messages [ 1370.098203] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.58@tcp (stopping) [ 1372.641584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1372.646318] Lustre: Skipped 1 previous similar message [ 1373.218854] Lustre: server umount lustre-MDT0000 complete [ 1375.188393] LustreError: 44996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.052505] LustreError: 44980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443687 with bad export cookie 14501196437493701705 [ 1377.069804] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1377.078267] LustreError: 44980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1377.760704] LustreError: 44998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.761050] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1377.808596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1380.300512] LustreError: 46010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.880494] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1382.895812] Lustre: Skipped 1 previous similar message [ 1383.584582] Lustre: server umount lustre-MDT0001 complete [ 1412.063918] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1412.384267] Lustre: server umount lustre-OST0000 complete [ 1437.848978] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 1441.863840] Key type lgssc unregistered [ 1442.285576] LNet: 48868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1442.294130] LNetError: 48868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.349101] LNet: Removed LNI 192.168.202.158@tcp [ 1444.165832] Key type .llcrypt unregistered [ 1444.177854] Key type ._llcrypt unregistered [ 1473.410802] Key type ._llcrypt registered [ 1473.419233] Key type .llcrypt registered [ 1473.528566] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 1475.012822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1475.060275] alg: No test for adler32 (adler32-zlib) [ 1476.173144] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1476.444852] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 1478.129254] Key type lgssc registered [ 1479.517825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1488.182689] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1488.208631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1503.713388] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1508.831527] LustreError: 50253:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 1535.583697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.184699] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1554.253580] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1718.939762] Lustre: server umount lustre-OST0000 complete [ 1735.486610] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 1737.796844] Key type lgssc unregistered [ 1738.003065] LNet: 51934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.014718] LNetError: 51934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.039599] LNet: Removed LNI 192.168.202.158@tcp [ 1738.872311] Key type .llcrypt unregistered [ 1738.874485] Key type ._llcrypt unregistered [ 1766.799560] Key type ._llcrypt registered [ 1766.801577] Key type .llcrypt registered [ 1766.892250] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 1767.946712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1768.120000] alg: No test for adler32 (adler32-zlib) [ 1769.246269] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1769.610880] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 1771.463288] Key type lgssc registered [ 1773.425734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1783.308350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1783.336647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.921469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.668443] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1794.019905] LustreError: 53363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1794.042651] LustreError: 53363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1799.150749] LustreError: 53364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1799.740665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.234621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1805.288680] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1813.670631] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1819.918853] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1827.423837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1828.015942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.057048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1834.321814] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1841.706977] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1844.704467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1844.731984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1845.799409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1845.816335] Lustre: Skipped 1 previous similar message [ 1850.854958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1850.866443] Lustre: Skipped 1 previous similar message [ 1855.972600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1855.978176] Lustre: Skipped 1 previous similar message [ 1858.015156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1858.210303] Lustre: server umount lustre-OST0000 complete [ 1866.210057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.215772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.218528] Lustre: Skipped 1 previous similar message [ 1866.232621] Lustre: Skipped 1 previous similar message [ 1868.733362] Lustre: server umount lustre-MDT0000 complete [ 1871.331441] LustreError: 53364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.356322] LustreError: 53364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1873.038015] LustreError: 53345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444183 with bad export cookie 14555052339942155613 [ 1873.047476] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1873.050329] LustreError: 53345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1873.505523] Lustre: server umount lustre-MDT0001 complete [ 1886.103404] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 1889.140621] Key type lgssc unregistered [ 1889.474874] LNet: 56900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.478870] LNetError: 56900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.499543] LNet: Removed LNI 192.168.202.158@tcp [ 1890.657619] Key type .llcrypt unregistered [ 1890.661195] Key type ._llcrypt unregistered [ 1915.185405] Key type ._llcrypt registered [ 1915.187689] Key type .llcrypt registered [ 1915.284443] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 1916.383569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1916.532870] alg: No test for adler32 (adler32-zlib) [ 1917.529613] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1917.810710] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 1919.617485] Key type lgssc registered [ 1920.950389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1931.089239] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1931.132096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1946.784173] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1951.909640] LustreError: 58288:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 1978.655924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.915801] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1992.350851] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1996.960544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.319842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.376509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2001.484319] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2002.402500] LustreError: 58314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.418835] LustreError: 58314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2007.526049] LustreError: 58313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.025443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.633349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2014.041165] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2022.373151] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2029.191291] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2033.128826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2033.154307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2033.172512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2034.147719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2034.153089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.788106] Lustre: server umount lustre-OST0000 complete [ 2064.864579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.898421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2064.901332] Lustre: Skipped 1 previous similar message [ 2068.948369] Lustre: server umount lustre-MDT0000 complete [ 2069.985817] LustreError: 59328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.567895] LustreError: 58295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444383 with bad export cookie 2853358555514387795 [ 2073.574348] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2073.585178] LustreError: 58295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2080.044543] Lustre: server umount lustre-MDT0001 complete [ 2093.825616] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2096.967552] Key type lgssc unregistered [ 2097.266188] LNet: 62040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.272919] LNetError: 62040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2097.293864] LNet: Removed LNI 192.168.202.158@tcp [ 2098.218756] Key type .llcrypt unregistered [ 2098.220596] Key type ._llcrypt unregistered [ 2120.566111] Key type ._llcrypt registered [ 2120.568106] Key type .llcrypt registered [ 2120.655508] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 2121.463114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2121.497071] alg: No test for adler32 (adler32-zlib) [ 2122.584787] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2122.771339] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 2124.455353] Key type lgssc registered [ 2125.626877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2133.191027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2133.216342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.620530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.412118] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2144.742312] LustreError: 63452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.771237] LustreError: 63452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2145.763086] LustreError: 63453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.316101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.745739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2153.742637] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2163.021856] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2170.760387] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2178.444367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2178.745449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2180.787158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2185.196550] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2191.693833] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2219.378931] Lustre: server umount lustre-OST0000 complete [ 2224.101754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.106205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2224.119927] Lustre: Skipped 1 previous similar message [ 2224.138451] Lustre: Skipped 1 previous similar message [ 2229.216878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2229.229539] Lustre: Skipped 1 previous similar message [ 2232.396414] Lustre: server umount lustre-MDT0000 complete [ 2234.341303] LustreError: 63449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.353670] LustreError: 63449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2236.506022] LustreError: 63433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444546 with bad export cookie 5882354194867756490 [ 2236.526412] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2246.300949] Lustre: server umount lustre-MDT0001 complete [ 2258.590382] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2261.370697] Key type lgssc unregistered [ 2261.599554] LNet: 67014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.604419] LNetError: 67014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2261.613697] LNet: Removed LNI 192.168.202.158@tcp [ 2263.041900] Key type .llcrypt unregistered [ 2263.043736] Key type ._llcrypt unregistered [ 2292.874530] Key type ._llcrypt registered [ 2292.875916] Key type .llcrypt registered [ 2293.031438] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 2294.413977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2294.448513] alg: No test for adler32 (adler32-zlib) [ 2295.543919] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2295.809785] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 2297.552515] Key type lgssc registered [ 2298.782302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2308.755299] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 10:10:17 (1788444617) [ 2315.730831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2315.752348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.282239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.316368] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2327.526492] LustreError: 68721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.536498] LustreError: 68721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2330.942048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.244699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2336.242376] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2345.156507] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2350.196778] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2356.263171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2356.623961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.564148] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2362.825308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2369.318904] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2379.519883] Lustre: Failing over lustre-MDT0000 [ 2379.971231] Lustre: server umount lustre-MDT0000 complete [ 2381.281459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2381.284703] LustreError: 70144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.289419] Lustre: Skipped 1 previous similar message [ 2381.309610] LustreError: 70144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2382.303374] LustreError: 70144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.409120] LustreError: 68721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.440362] LustreError: 68721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2391.266254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2391.456112] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2391.521537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2391.526214] Lustre: Skipped 1 previous similar message [ 2391.626754] Lustre: *** cfs_fail_loc=135, val=0*** [ 2391.773917] LustreError: 71811:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2391.784306] LustreError: 71811:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.158@tcp: cfg command failed: rc = -2 [ 2391.800287] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2391.800287] [ 2391.823586] LustreError: MGC192.168.202.158@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2391.844082] LustreError: 71799:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2391.869030] LustreError: 71799:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2391.884950] LustreError: 71799:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2391.894527] LustreError: 71799:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2391.923681] Lustre: server umount lustre-MDT0000 complete [ 2391.928556] LustreError: 71799:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2392.530174] LustreError: 68718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.569768] LustreError: 68718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2399.137690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2399.311702] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2399.318854] LustreError: 68713:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2399.746608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.846154] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2402.762828] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2404.401080] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2404.851614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2404.877712] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2404.954633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2412.000591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2412.007519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.030545] Lustre: Skipped 1 previous similar message [ 2412.035080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2412.041530] Lustre: Skipped 1 previous similar message [ 2415.073220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.083259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2417.615847] Lustre: server umount lustre-OST0000 complete [ 2425.314105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.314853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2425.327698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.327709] Lustre: Skipped 1 previous similar message [ 2425.330728] Lustre: Skipped 1 previous similar message [ 2427.394325] Lustre: server umount lustre-MDT0000 complete [ 2430.371527] LustreError: 68701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444740 with bad export cookie 14916145867974865674 [ 2430.374353] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.381677] LustreError: 68701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2430.734779] Lustre: server umount lustre-MDT0001 complete [ 2438.937461] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2442.066696] Key type lgssc unregistered [ 2442.383920] LNet: 73591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2442.390238] LNetError: 73591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2442.417518] LNet: Removed LNI 192.168.202.158@tcp [ 2443.499163] Key type .llcrypt unregistered [ 2443.502084] Key type ._llcrypt unregistered [ 2471.831433] Key type ._llcrypt registered [ 2471.834905] Key type .llcrypt registered [ 2471.941882] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 2472.835184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2472.868065] alg: No test for adler32 (adler32-zlib) [ 2474.186989] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2474.548664] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 2476.234698] Key type lgssc registered [ 2477.910551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2486.843980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2486.902564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.572556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.427482] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2498.017794] LustreError: 75009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.033503] LustreError: 75009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2503.138462] LustreError: 75008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.171865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.458703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2508.449098] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2515.791736] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2522.295744] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2529.485444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2529.891046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.017302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2536.490909] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2544.796286] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2554.875486] Lustre: Failing over lustre-MDT0000 [ 2555.081935] Lustre: server umount lustre-MDT0000 complete [ 2557.950744] LustreError: 75003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.946452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.972291] Lustre: Skipped 2 previous similar messages [ 2563.026738] LustreError: 75005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.067825] LustreError: 75005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2565.010911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.144582] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2565.266450] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2565.381411] LustreError: 78096:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2565.386351] LustreError: 78096:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.158@tcp: cfg command failed: rc = -9 [ 2565.391661] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2565.391661] [ 2565.401486] LustreError: MGC192.168.202.158@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2565.416344] LustreError: 78084:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2565.423980] LustreError: 78084:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2565.432888] LustreError: 78084:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2565.436809] LustreError: 78084:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2565.455457] Lustre: server umount lustre-MDT0000 complete [ 2565.460244] LustreError: 78084:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2568.143216] LustreError: 76848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.164391] LustreError: 76848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2572.384591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.520479] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2572.921520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.007501] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2573.260700] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2577.902962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2577.927913] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2577.989414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2578.923887] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2588.132225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.156733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.155559] Lustre: server umount lustre-OST0000 complete [ 2598.368290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2598.369108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2598.381859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2598.387360] Lustre: Skipped 2 previous similar messages [ 2598.391098] Lustre: Skipped 1 previous similar message [ 2602.637915] Lustre: server umount lustre-MDT0000 complete [ 2603.488354] LustreError: 76393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.522482] LustreError: 76393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2606.790960] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2606.791523] LustreError: 74990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444916 with bad export cookie 900280953950804044 [ 2606.821149] LustreError: 74990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2607.491348] Lustre: server umount lustre-MDT0001 complete [ 2620.451569] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2624.827854] Key type lgssc unregistered [ 2625.390704] LNet: 79878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2625.405093] LNetError: 79878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2625.449100] LNet: Removed LNI 192.168.202.158@tcp [ 2626.703358] Key type .llcrypt unregistered [ 2626.705085] Key type ._llcrypt unregistered [ 2644.609863] Key type ._llcrypt registered [ 2644.611889] Key type .llcrypt registered [ 2644.714071] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2645.010412] Key type .llcrypt unregistered [ 2645.011589] Key type ._llcrypt unregistered [ 2673.933079] Key type ._llcrypt registered [ 2673.934850] Key type .llcrypt registered [ 2674.006103] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 2674.989962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2675.020905] alg: No test for adler32 (adler32-zlib) [ 2676.202716] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2676.495112] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 2678.231235] Key type lgssc registered [ 2679.921092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2689.906537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2689.927855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.510632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.923550] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2701.798389] LustreError: 81787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.815695] LustreError: 81787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2706.915230] LustreError: 81788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.673534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.276706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2713.817214] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2722.822334] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2728.648810] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2736.440976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2736.866326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.853813] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2752.146951] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2761.466494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2771.936793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2771.949969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.959906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.964158] Lustre: Skipped 1 previous similar message [ 2776.507130] Lustre: server umount lustre-OST0000 complete [ 2782.178251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.188556] Lustre: Skipped 1 previous similar message [ 2782.196806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.204854] Lustre: Skipped 1 previous similar message [ 2786.752685] Lustre: server umount lustre-MDT0000 complete [ 2787.296462] LustreError: 81782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.322467] LustreError: 81782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2790.868773] LustreError: 81767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445100 with bad export cookie 7839520901999320499 [ 2790.874455] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2790.887338] LustreError: 81767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2791.253703] Lustre: server umount lustre-MDT0001 complete [ 2802.907374] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2806.156940] Key type lgssc unregistered [ 2806.445790] LNet: 85345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2806.455288] LNetError: 85345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2806.481344] LNet: Removed LNI 192.168.202.158@tcp [ 2807.564580] Key type .llcrypt unregistered [ 2807.566196] Key type ._llcrypt unregistered [ 2837.035429] Key type ._llcrypt registered [ 2837.056407] Key type .llcrypt registered [ 2837.276828] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 2839.093687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2839.451196] alg: No test for adler32 (adler32-zlib) [ 2840.636819] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2840.875561] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 2842.647300] Key type lgssc registered [ 2843.973841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2850.989755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2851.018877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.434483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.243266] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2862.561035] LustreError: 86758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.572411] LustreError: 86758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2865.724070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.185083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2870.796322] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2878.694699] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2885.580954] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2892.966335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2893.346239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.468070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2899.806507] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2907.942337] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2917.856792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2917.860327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2917.865869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2921.957148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2921.959104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2921.970039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2927.079556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2927.089276] Lustre: Skipped 2 previous similar messages [ 2932.192692] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2932.195037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2932.211493] Lustre: Skipped 1 previous similar message [ 2932.633497] Lustre: server umount lustre-OST0000 complete [ 2937.317659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.326026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2937.370724] Lustre: Skipped 1 previous similar message [ 2937.391097] Lustre: Skipped 1 previous similar message [ 2943.339281] Lustre: server umount lustre-MDT0000 complete [ 2947.557236] LustreError: 87485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.612166] LustreError: 87485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2948.707918] LustreError: 89336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445258 with bad export cookie 10340114638375781212 [ 2948.713193] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2948.715407] LustreError: 89336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2949.401459] Lustre: server umount lustre-MDT0001 complete [ 2962.678490] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2966.031481] Key type lgssc unregistered [ 2966.469040] LNet: 90295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2966.490776] LNetError: 90295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2967.530856] LNet: Removed LNI 192.168.202.158@tcp [ 2968.217073] Key type .llcrypt unregistered [ 2968.229151] Key type ._llcrypt unregistered [ 2997.395789] Key type ._llcrypt registered [ 2997.399377] Key type .llcrypt registered [ 2997.530222] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 2998.684834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2998.792632] alg: No test for adler32 (adler32-zlib) [ 2999.843502] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3000.024347] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 3001.673263] Key type lgssc registered [ 3003.036364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3011.843888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3011.861504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.283041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3017.618961] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3023.842640] LustreError: 91710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.852140] LustreError: 91710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3026.203061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.606781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3030.683524] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3038.064394] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3044.465516] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3052.673986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3053.137930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.288767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3061.114515] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3071.082200] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3083.749387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3083.754092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3083.764848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3087.328402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3087.334328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3087.350497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3087.367221] Lustre: Skipped 1 previous similar message [ 3089.490409] Lustre: server umount lustre-OST0000 complete [ 3097.573432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.598846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.616677] Lustre: Skipped 1 previous similar message [ 3102.689986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3102.700040] Lustre: Skipped 1 previous similar message [ 3107.809422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3107.815119] Lustre: Skipped 1 previous similar message [ 3108.319167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3110.551953] Lustre: server umount lustre-MDT0000 complete [ 3112.928493] LustreError: 91709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3112.946298] LustreError: 91709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3114.912452] LustreError: 91690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445424 with bad export cookie 2031386497962620513 [ 3114.917133] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3114.919021] LustreError: 91690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3115.372262] Lustre: server umount lustre-MDT0001 complete [ 3125.838926] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 3128.960418] Key type lgssc unregistered [ 3129.385134] LNet: 95265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3129.393426] LNetError: 95265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3129.418251] LNet: Removed LNI 192.168.202.158@tcp [ 3130.542687] Key type .llcrypt unregistered [ 3130.544299] Key type ._llcrypt unregistered [ 3164.358894] Key type ._llcrypt registered [ 3164.360989] Key type .llcrypt registered [ 3164.513540] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 3165.965653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3166.083384] alg: No test for adler32 (adler32-zlib) [ 3167.140953] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3167.392506] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 3169.080596] Key type lgssc registered [ 3170.810583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3181.097863] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3181.119943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3196.703678] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3201.823463] LustreError: 96655:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 3228.639713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3235.590727] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3243.618178] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3249.589421] Lustre: server umount lustre-OST0000 complete [ 3259.452918] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:26:08 (1788445568) [ 3265.962812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.433102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.901387] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3271.663110] LustreError: 98294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.693846] LustreError: 98294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3276.776812] LustreError: 98293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3280.058198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3280.496733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3286.351755] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3296.322196] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3303.558101] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3311.109996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3311.355381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3312.432320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3319.162612] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3328.151832] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3341.288154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3341.293504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3341.315329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3346.402449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3346.408996] Lustre: Skipped 1 previous similar message [ 3347.323641] Lustre: server umount lustre-OST0000 complete [ 3352.032487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3352.039257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.048809] Lustre: Skipped 1 previous similar message [ 3352.058602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.076368] Lustre: Skipped 1 previous similar message [ 3356.644543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3356.645605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3356.668776] Lustre: Skipped 1 previous similar message [ 3361.763426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3361.771249] Lustre: Skipped 1 previous similar message [ 3366.374147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3366.886294] LustreError: 98288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.899433] LustreError: 98288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3368.571802] Lustre: server umount lustre-MDT0000 complete [ 3371.001479] LustreError: 101032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.031271] LustreError: 101032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3372.600498] LustreError: 98274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445682 with bad export cookie 15115692203953889992 [ 3372.602333] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3372.614815] LustreError: 98274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3373.069750] Lustre: server umount lustre-MDT0001 complete [ 3386.442685] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 3389.719509] Key type lgssc unregistered [ 3390.086563] LNet: 101992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3390.097841] LNetError: 101992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3391.143971] LNet: Removed LNI 192.168.202.158@tcp [ 3392.159373] Key type .llcrypt unregistered [ 3392.169257] Key type ._llcrypt unregistered [ 3438.291463] Key type ._llcrypt registered [ 3438.293967] Key type .llcrypt registered [ 3438.389349] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 3439.596937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3439.714482] alg: No test for adler32 (adler32-zlib) [ 3440.761593] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3441.161976] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 3442.831838] Key type lgssc registered [ 3444.038815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3451.883583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3451.902753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.335878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3456.557489] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3463.649605] LustreError: 104181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3463.666416] LustreError: 104181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3463.828462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.063153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3467.466891] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3476.612221] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3483.375198] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3491.894096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3492.472987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.623293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3500.721922] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3508.727123] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3519.459904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3519.465760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3519.472599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3520.482437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3520.497947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3520.499784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3520.530548] Lustre: Skipped 1 previous similar message [ 3522.990918] Lustre: server umount lustre-OST0000 complete [ 3530.721324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3530.723960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3530.732451] Lustre: Skipped 1 previous similar message [ 3530.748635] Lustre: Skipped 1 previous similar message [ 3532.820148] Lustre: server umount lustre-MDT0000 complete [ 3535.840070] LustreError: 104909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3535.861770] LustreError: 104909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3536.527491] LustreError: 104162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445846 with bad export cookie 16582838490513149518 [ 3536.532576] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3536.542291] LustreError: 104162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3536.979140] Lustre: server umount lustre-MDT0001 complete [ 3549.456100] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 3552.977934] Key type lgssc unregistered [ 3553.268815] LNet: 107718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3553.275313] LNetError: 107718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3554.341904] LNet: Removed LNI 192.168.202.158@tcp [ 3555.353919] Key type .llcrypt unregistered [ 3555.364415] Key type ._llcrypt unregistered [ 3574.344479] Key type ._llcrypt registered [ 3574.347241] Key type .llcrypt registered [ 3574.493859] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 3575.765727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3575.822480] alg: No test for adler32 (adler32-zlib) [ 3577.111875] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3577.497764] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 3579.167202] Key type lgssc registered [ 3580.543730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3588.556586] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3588.580260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3604.192262] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3609.313724] LustreError: 108962:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 3635.999371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.736430] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3649.964917] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3655.626868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.871229] LustreError: MGC192.168.202.158@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3655.878723] LustreError: 109986:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3655.882757] LustreError: 109986:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3655.889361] LustreError: 109986:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3655.978165] Lustre: server umount lustre-MDT0000 complete [ 3655.979932] LustreError: 109986:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3676.129513] Lustre: 108300:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445970/real 1788445970] req@0000000091599001 x1875321434410368/t0(0) o400->MGC192.168.202.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788445986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3676.193693] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3676.217721] LustreError: 108974:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3696.765958] Lustre: server umount lustre-OST0000 complete [ 3706.257606] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_hostid [ 3716.833827] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 3752.410466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.761929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3752.825977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3752.908708] Lustre: lustre-MDT0000: new disk, initializing [ 3753.020919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3753.034722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3757.715575] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3768.050656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3768.126805] Lustre: 112781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3768.227675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3768.231099] Lustre: Skipped 1 previous similar message [ 3768.336446] Lustre: lustre-MDT0001: new disk, initializing [ 3768.437621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3768.490094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3768.504146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3772.918441] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3784.642393] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3791.263744] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3800.253425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3800.425710] Lustre: lustre-OST0000: new disk, initializing [ 3800.427912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3800.431178] Lustre: 114750:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3800.515034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.995704] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3809.994826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3810.035381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3810.329934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3819.620533] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3825.508950] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3825.740746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3831.665689] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3831.984667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3835.873547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.880738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3835.887692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3835.909600] Lustre: Skipped 1 previous similar message [ 3840.512688] Lustre: server umount lustre-OST0000 complete [ 3844.076981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3844.089942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3844.097225] Lustre: Skipped 2 previous similar messages [ 3846.113075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3846.121626] Lustre: Skipped 1 previous similar message [ 3850.426649] Lustre: server umount lustre-MDT0000 complete [ 3851.244575] LustreError: 112792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3851.271472] LustreError: 112792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3854.750938] LustreError: 112772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446164 with bad export cookie 12460563607269143702 [ 3854.755790] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3854.763810] LustreError: 112772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3855.606363] Lustre: server umount lustre-MDT0001 complete [ 3864.282882] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:36:13 (1788446173) [ 3867.715838] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3884.965966] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_hostid [ 3894.962593] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 3931.929375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3932.242642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3932.285952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3932.378244] Lustre: lustre-MDT0000: new disk, initializing [ 3932.500064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3932.527429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3937.483208] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3948.124246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3948.216662] Lustre: 119662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3948.253551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3948.255912] Lustre: Skipped 1 previous similar message [ 3948.309402] Lustre: lustre-MDT0001: new disk, initializing [ 3948.381058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3948.411421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3948.420570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3953.294966] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3962.830990] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3968.597350] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3975.682943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3976.012252] Lustre: lustre-OST0000: new disk, initializing [ 3976.015573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3976.022655] Lustre: 121629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3976.104318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.453146] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3983.953927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3983.969641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3984.121144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3993.114559] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3998.374616] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3998.612644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4004.204904] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4004.461946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4009.956112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4009.957740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.964847] LustreError: Skipped 2 previous similar messages [ 4009.975547] Lustre: Skipped 1 previous similar message [ 4009.977305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4012.983691] Lustre: server umount lustre-OST0000 complete [ 4020.192225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.194927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.216935] Lustre: Skipped 2 previous similar messages [ 4020.240854] Lustre: Skipped 2 previous similar messages [ 4023.023043] Lustre: server umount lustre-MDT0000 complete [ 4025.314868] LustreError: 119668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4025.353360] LustreError: 119668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4026.925873] LustreError: 119653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446336 with bad export cookie 12460563607269144584 [ 4026.934962] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4027.236526] Lustre: server umount lustre-MDT0001 complete [ 4035.976899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4036.254437] LustreError: 123846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4036.326455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4041.701684] LustreError: 123847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4042.058207] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4046.827433] LustreError: 123846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.774123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4058.526435] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4067.465317] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4074.971206] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4082.227026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4089.463460] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4098.056880] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4115.179600] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4121.569961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.608305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4121.623224] Lustre: Skipped 1 previous similar message [ 4121.982515] Lustre: server umount lustre-OST0000 complete [ 4132.685513] Lustre: server umount lustre-MDT0000 complete [ 4136.317811] LustreError: 123829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446446 with bad export cookie 12460563607269145466 [ 4136.328692] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4136.334070] LustreError: 123829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4136.957791] Lustre: server umount lustre-MDT0001 complete [ 4151.493297] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 4155.576989] Key type lgssc unregistered [ 4155.844385] LNet: 127506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4155.854358] LNetError: 127506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4155.901286] LNet: Removed LNI 192.168.202.158@tcp [ 4156.817612] Key type .llcrypt unregistered [ 4156.824752] Key type ._llcrypt unregistered [ 4175.760289] Key type ._llcrypt registered [ 4175.763605] Key type .llcrypt registered [ 4175.919287] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_hostid [ 4190.595303] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 4192.706538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4192.746771] alg: No test for adler32 (adler32-zlib) [ 4193.820603] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4194.154485] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 4195.832546] Key type lgssc registered [ 4197.165818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4234.672368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4234.710036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4235.990442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4236.028669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4236.143797] Lustre: lustre-MDT0000: new disk, initializing [ 4236.229532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4236.250205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4242.323231] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4253.794389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4253.889104] Lustre: 130661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4253.940142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4253.942782] Lustre: Skipped 1 previous similar message [ 4253.998440] Lustre: lustre-MDT0001: new disk, initializing [ 4254.085941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4254.144093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4254.160448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4258.982394] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4268.660898] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4274.934541] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4282.241346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4282.494356] Lustre: lustre-OST0000: new disk, initializing [ 4282.497873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4282.502270] Lustre: 132627:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4282.590328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.332089] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4291.690287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4291.712317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4291.883427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4299.613752] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4304.734770] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4304.942667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4310.041162] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4310.247988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4312.546967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4312.558105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4312.561299] LustreError: Skipped 1 previous similar message [ 4312.561949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4312.593174] Lustre: Skipped 1 previous similar message [ 4317.669411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4317.676156] Lustre: Skipped 2 previous similar messages [ 4318.597972] Lustre: server umount lustre-OST0000 complete [ 4322.786291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.786291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.787287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4322.832984] Lustre: Skipped 1 previous similar message [ 4327.905097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4327.913676] Lustre: Skipped 1 previous similar message [ 4328.919199] Lustre: server umount lustre-MDT0000 complete [ 4332.473572] LustreError: 130653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446642 with bad export cookie 11957291859482744894 [ 4332.497155] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4332.943452] Lustre: server umount lustre-MDT0001 complete [ 4341.752852] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:44:10 (1788446650) [ 4348.905046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.625962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.489430] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4355.040252] LustreError: 135040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4355.063660] LustreError: 135040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4359.137746] LustreError: 135039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4363.867348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.370787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4369.973603] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4377.958581] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4383.535662] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4389.857747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4389.865252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.890513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.899316] Lustre: Skipped 1 previous similar message [ 4400.095613] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4400.097500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4400.124289] Lustre: Skipped 3 previous similar messages [ 4400.430312] Lustre: server umount lustre-MDT0000 complete [ 4405.107478] LustreError: 135020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446715 with bad export cookie 11957291859482745776 [ 4405.118036] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4405.126520] LustreError: 135020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4405.558188] Lustre: server umount lustre-MDT0001 complete [ 4415.157254] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:45:23 (1788446723) [ 4422.054666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4437.727799] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4442.847985] LustreError: 137458:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 4468.575290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4474.723316] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4481.655687] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4483.759852] Lustre: server umount lustre-OST0000 complete [ 4492.678622] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:46:41 (1788446801) [ 4499.089562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4499.446879] LustreError: 138903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.538100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4503.809224] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4504.545876] LustreError: 138904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.670856] LustreError: 138903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4512.617182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.365216] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4524.841461] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4530.663901] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4538.798940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4539.219296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4539.233213] Lustre: Skipped 1 previous similar message [ 4545.964786] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4553.496527] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4567.054335] Lustre: server umount lustre-MDT0000 complete [ 4568.040195] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.041977] LustreError: 138903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4568.070317] Lustre: Skipped 2 previous similar messages [ 4570.711493] LustreError: 138885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446880 with bad export cookie 11957291859482746434 [ 4570.720257] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4571.137418] Lustre: server umount lustre-MDT0001 complete [ 4584.906976] Lustre: server umount lustre-OST0000 complete [ 4593.485083] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:48:22 (1788446902) [ 4599.999561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4600.689060] LustreError: 142556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4600.712922] LustreError: 142556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4605.850619] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4613.720176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.141485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4614.154245] Lustre: Skipped 1 previous similar message [ 4618.122412] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4626.431665] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4632.606447] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4639.293228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4645.760157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4646.283975] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4653.362617] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4658.348418] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4658.591809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4664.081636] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4664.356825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4666.346966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4666.348737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4666.363415] LustreError: Skipped 1 previous similar message [ 4666.381354] Lustre: Skipped 1 previous similar message [ 4666.389046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4666.409724] Lustre: Skipped 1 previous similar message [ 4672.640905] Lustre: server umount lustre-OST0000 complete [ 4676.578918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4676.618755] Lustre: Skipped 1 previous similar message [ 4682.755667] Lustre: server umount lustre-MDT0000 complete [ 4686.468327] LustreError: 142537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446996 with bad export cookie 11957291859482747505 [ 4686.485465] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4686.487894] LustreError: 142537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4686.759890] Lustre: server umount lustre-MDT0001 complete [ 4694.088687] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:50:03 (1788447003) [ 4700.506113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4716.001219] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4721.124521] LustreError: 146713:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 4746.912173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4746.920094] Lustre: Skipped 1 previous similar message [ 4754.225935] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4763.231944] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4770.630062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.927120] LustreError: 146738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.945579] LustreError: 146738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4771.066250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4775.116724] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4784.318367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.982474] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4798.468494] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4806.160405] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4811.845316] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4812.024170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4816.076385] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4816.278461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4820.965578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4820.966879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4820.968346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4820.968353] Lustre: Skipped 6 previous similar messages [ 4820.979354] Lustre: Skipped 2 previous similar messages [ 4824.267714] Lustre: server umount lustre-MDT0000 complete [ 4827.833433] LustreError: 146720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447137 with bad export cookie 11957291859482748534 [ 4827.848457] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4828.121178] Lustre: server umount lustre-MDT0001 complete [ 4841.859326] Lustre: server umount lustre-OST0000 complete [ 4850.092685] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:52:39 (1788447159) [ 4856.370039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4872.032180] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4877.152119] LustreError: 150907:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 4909.977514] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4918.503698] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4925.514841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4926.156723] LustreError: 150932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.169804] LustreError: 150932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4926.289630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4931.095520] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4940.475634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4945.778770] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4953.585546] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4958.782328] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4965.420205] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4965.602615] Lustre: lustre-OST0001: new disk, initializing [ 4965.605340] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4965.613367] Lustre: 153778:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4971.736573] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4974.724175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4974.738312] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4974.965549] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4981.327332] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4985.989160] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4986.269960] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4990.897695] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4991.062805] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4992.994807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4993.003716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4993.010498] Lustre: Skipped 1 previous similar message [ 4993.017673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4993.027599] Lustre: Skipped 2 previous similar messages [ 5007.327243] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5007.418636] Lustre: server umount lustre-OST0000 complete [ 5016.035497] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5016.043457] LustreError: Skipped 1 previous similar message [ 5026.271202] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5026.455421] Lustre: server umount lustre-OST0001 complete [ 5044.191179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5044.744305] Lustre: server umount lustre-MDT0000 complete [ 5049.540711] LustreError: 150915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447359 with bad export cookie 11957291859482749486 [ 5049.549078] LustreError: 150915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5049.551752] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5049.985652] Lustre: server umount lustre-MDT0001 complete [ 5075.744441] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:56:23 (1788447383) [ 5077.594235] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5079.223170] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5081.428130] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:56:29 (1788447389) [ 5089.619662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.885874] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5089.919930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5090.204419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5090.218240] Lustre: Skipped 6 previous similar messages [ 5094.844994] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5103.516080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5103.716996] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5103.811495] Lustre: 157347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5110.482635] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5118.123589] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5123.495578] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5130.164448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5130.424901] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5136.185865] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5139.038722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5142.503177] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5145.753532] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5145.937736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5150.995225] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5151.220185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5154.279352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5154.292174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5154.306363] Lustre: Skipped 6 previous similar messages [ 5154.331708] Lustre: Skipped 19 previous similar messages [ 5159.776340] Lustre: server umount lustre-OST0000 complete [ 5183.235164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5184.808636] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.824629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.828988] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5191.126710] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5199.087543] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5204.428258] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5204.674326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5209.992331] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5210.225180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5216.392345] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f95074a7000.ost_server_uuid 50 [ 5217.979690] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f95074a7000.ost_server_uuid in FULL state after 0 sec [ 5218.223845] LustreError: 157352:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.58@tcp) returned error from blocking AST (req@000000000cdf48ae x1875322081645952 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000c1d42196/0xa5f0d5980023b388 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.58@tcp remote: 0xe74e0ae2beb9c287 expref: 5 pid: 157354 timeout: 5318 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5218.291234] LustreError: lustre-MDT0000: A client on nid 192.168.202.58@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5218.309983] LustreError: 157342:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.58@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000c1d42196/0xa5f0d5980023b388 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.58@tcp remote: 0xe74e0ae2beb9c287 expref: 6 pid: 157354 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5223.922232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5229.130889] Lustre: server umount lustre-OST0000 complete [ 5239.943165] Lustre: server umount lustre-MDT0000 complete [ 5241.312719] LustreError: 157354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5241.353661] LustreError: 157354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 5245.207069] LustreError: 157338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447555 with bad export cookie 11957291859482750669 [ 5245.211883] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5245.222484] LustreError: 157338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5251.661316] Lustre: server umount lustre-MDT0001 complete [ 5263.207789] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5266.625592] Key type lgssc unregistered [ 5267.167556] LNet: 163465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5267.182704] LNetError: 163465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5267.245465] LNet: Removed LNI 192.168.202.158@tcp [ 5268.533375] Key type .llcrypt unregistered [ 5268.536292] Key type ._llcrypt unregistered [ 5295.415440] Key type ._llcrypt registered [ 5295.417217] Key type .llcrypt registered [ 5295.496200] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 5296.533849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5296.555268] alg: No test for adler32 (adler32-zlib) [ 5297.996563] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5298.286028] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 5299.975242] Key type lgssc registered [ 5301.367292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5312.196616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5312.223905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5313.593164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5319.762492] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5323.753995] LustreError: 164880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.798497] LustreError: 164880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5328.046240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.294470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5333.495759] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5341.708874] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5346.808997] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5356.426339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5356.760618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5358.914956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5368.953313] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5380.168628] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5387.125598] Lustre: Failing over lustre-MDT0000 [ 5388.095190] Lustre: server umount lustre-MDT0000 complete [ 5388.294238] LustreError: 164876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5388.812513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5388.839986] Lustre: Skipped 1 previous similar message [ 5393.894209] LustreError: 164874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5393.907390] LustreError: 164874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5399.017525] LustreError: 164879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5399.035909] LustreError: 164879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5401.307330] LustreError: 167470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.58@tcp arrived at 1788447711 with bad export cookie 12616056693979712421 [ 5401.327642] LustreError: 167470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5404.064142] Lustre: 164192:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447698/real 1788447698] req@000000003d3d8e1a x1875323239035008/t0(0) o400->MGC192.168.202.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788447714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5404.112690] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5404.154675] LustreError: 164874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.183041] LustreError: 164874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5409.352796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5414.372667] LustreError: 167333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5414.381824] LustreError: 167333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5415.393689] LustreError: 164188:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000dc6b0314 x1875323239039872/t0(0) o250->MGC192.168.202.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5415.904427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.982734] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5421.027842] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5421.053627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5421.099854] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5423.883671] Lustre: lustre-MDT0000: Denying connection for new client 4dcc64ec-fcd2-4092-8746-532b0bad7cc9 (at 192.168.202.58@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5434.856904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5434.868114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.894209] Lustre: Skipped 1 previous similar message [ 5434.901903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5436.385664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5436.398482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.611198] Lustre: server umount lustre-OST0000 complete [ 5443.188701] LustreError: 168625:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5443.193477] Lustre: 167974:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5443.205429] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5443.243306] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5443.256162] LustreError: 167974:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a8e70046 x1875323239055360/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5443.292689] LustreError: 167974:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5443.312476] LustreError: 167974:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5443.345961] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5443.755222] Lustre: server umount lustre-MDT0000 complete [ 5446.625318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.662309] LustreError: 164876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.680673] LustreError: 164876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5447.760483] LustreError: 164861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447757 with bad export cookie 12616056693979712519 [ 5447.769576] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5447.769995] LustreError: 164188:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5447.784738] LustreError: 164861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5450.349281] Lustre: server umount lustre-MDT0001 complete [ 5463.531757] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5466.479880] Key type lgssc unregistered [ 5466.804876] LNet: 169379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5466.809648] LNetError: 169379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5466.827158] LNet: Removed LNI 192.168.202.158@tcp [ 5468.001219] Key type .llcrypt unregistered [ 5468.003816] Key type ._llcrypt unregistered [ 5492.702631] Key type ._llcrypt registered [ 5492.704163] Key type .llcrypt registered [ 5492.818391] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 5493.812838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5493.914937] alg: No test for adler32 (adler32-zlib) [ 5494.927751] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5495.139034] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 5496.799268] Key type lgssc registered [ 5498.259479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5506.660235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5506.679356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5508.244790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5513.218857] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5518.829559] LustreError: 170797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.852635] LustreError: 170797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5521.902696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.322394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5526.890354] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5535.741880] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5544.676656] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5555.957946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5556.306470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5564.010523] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5571.557205] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5578.880781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5589.473549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5589.481805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.500642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5589.526634] Lustre: Skipped 1 previous similar message [ 5589.547802] Lustre: Skipped 1 previous similar message [ 5594.598192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5595.751593] Lustre: server umount lustre-OST0000 complete [ 5604.320785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5604.342181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5604.379397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5604.389334] Lustre: Skipped 1 previous similar message [ 5606.233237] Lustre: server umount lustre-MDT0000 complete [ 5609.952586] LustreError: 170793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5609.989521] LustreError: 170793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5610.776342] LustreError: 170779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447920 with bad export cookie 11135770633617654264 [ 5610.791504] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5611.733509] Lustre: server umount lustre-MDT0001 complete [ 5626.334282] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5629.659755] Key type lgssc unregistered [ 5630.104731] LNet: 174355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5630.114629] LNetError: 174355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5630.134243] LNet: Removed LNI 192.168.202.158@tcp [ 5631.558557] Key type .llcrypt unregistered [ 5631.565524] Key type ._llcrypt unregistered [ 5671.003368] Key type ._llcrypt registered [ 5671.009668] Key type .llcrypt registered [ 5671.231347] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 5672.850656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5673.191194] alg: No test for adler32 (adler32-zlib) [ 5674.250484] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5674.621761] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 5676.327368] Key type lgssc registered [ 5677.866788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5687.730858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5687.777977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5689.180504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5692.957049] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5699.561049] LustreError: 176062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5699.574836] LustreError: 176062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5703.540048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5704.091741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5708.699240] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5716.251845] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5724.125938] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5732.608909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5733.037340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.089252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5739.480268] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5746.277272] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5758.957789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5758.966226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5758.980321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5759.972104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.976570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5759.978705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5759.988597] Lustre: Skipped 1 previous similar message [ 5763.769704] Lustre: server umount lustre-OST0000 complete [ 5770.215294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.222438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5770.230428] Lustre: Skipped 1 previous similar message [ 5770.247038] Lustre: Skipped 1 previous similar message [ 5774.250547] Lustre: server umount lustre-MDT0000 complete [ 5775.337161] LustreError: 176063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.355444] LustreError: 176063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5777.914643] LustreError: 176043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448087 with bad export cookie 2676242144038490429 [ 5777.925229] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5777.925874] LustreError: 176043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5778.194921] Lustre: server umount lustre-MDT0001 complete [ 5789.687661] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5792.584415] Key type lgssc unregistered [ 5792.887051] LNet: 179600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5792.906442] LNetError: 179600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5792.917396] LNet: Removed LNI 192.168.202.158@tcp [ 5793.903255] Key type .llcrypt unregistered [ 5793.905306] Key type ._llcrypt unregistered [ 5821.985354] Key type ._llcrypt registered [ 5821.987240] Key type .llcrypt registered [ 5822.063677] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 5823.187602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5823.259289] alg: No test for adler32 (adler32-zlib) [ 5824.432481] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5824.717539] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 5826.375299] Key type lgssc registered [ 5827.568831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5838.473720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5838.504909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5840.010324] Lustre: *** cfs_fail_loc=135, val=0*** [ 5840.184644] LustreError: 181130:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5840.209398] LustreError: 181130:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.158@tcp: cfg command failed: rc = -2 [ 5840.222940] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5840.222940] [ 5840.260762] LustreError: MGC192.168.202.158@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5840.285923] LustreError: 181091:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5840.301024] LustreError: 181091:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5840.308537] LustreError: 181091:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5840.312917] LustreError: 181091:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5840.422017] Lustre: server umount lustre-MDT0000 complete [ 5840.423734] LustreError: 181091:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5848.321618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5848.977919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5856.136054] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5858.274527] LustreError: 181503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5858.287813] LustreError: 181503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5863.402555] LustreError: 181502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.515192] LustreError: 181503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.705428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5868.879112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5873.296265] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5882.180940] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5888.438418] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5893.217407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5893.218888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5893.239269] Lustre: Skipped 1 previous similar message [ 5893.257401] Lustre: Skipped 1 previous similar message [ 5896.438708] Lustre: server umount lustre-MDT0000 complete [ 5898.724344] LustreError: 181502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5898.745335] LustreError: 181502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5900.597463] LustreError: 181482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448210 with bad export cookie 1130136423792765721 [ 5900.599612] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5900.609543] LustreError: 181482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5901.044453] Lustre: server umount lustre-MDT0001 complete [ 5913.549084] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5916.812125] Key type lgssc unregistered [ 5917.277503] LNet: 183830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5917.289023] LNetError: 183830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5917.314177] LNet: Removed LNI 192.168.202.158@tcp [ 5918.328921] Key type .llcrypt unregistered [ 5918.331213] Key type ._llcrypt unregistered [ 5938.175930] Key type ._llcrypt registered [ 5938.177544] Key type .llcrypt registered [ 5938.266736] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5938.563463] Key type .llcrypt unregistered [ 5938.565384] Key type ._llcrypt unregistered [ 5958.883238] Key type ._llcrypt registered [ 5958.885756] Key type .llcrypt registered [ 5959.037262] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 5961.178334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5961.216294] alg: No test for adler32 (adler32-zlib) [ 5962.323432] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5962.608627] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 5964.375315] Key type lgssc registered [ 5965.640495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5973.299489] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5973.316515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5989.024917] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5994.143625] LustreError: 185967:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 6020.895236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6028.367697] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6036.964583] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6044.130323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6044.933636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6045.034730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6050.279424] LustreError: 185992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6050.301936] LustreError: 185992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6050.472693] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6055.412255] LustreError: 185993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6055.446686] LustreError: 185993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6060.447764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.518054] LustreError: 187004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6060.545373] LustreError: 187004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6061.157413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6066.302119] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6074.208825] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6081.206612] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6084.478337] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6101.479515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6101.486649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.503193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6101.991457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6102.003480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6102.029141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6103.403858] Lustre: server umount lustre-OST0000 complete [ 6112.238303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6112.248091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6112.268583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6113.580797] Lustre: server umount lustre-MDT0000 complete [ 6117.345137] LustreError: 187009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6117.390115] LustreError: 187009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6117.853333] LustreError: 185975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448427 with bad export cookie 13890925599313154379 [ 6117.855963] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6117.870549] LustreError: 185975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6118.286517] Lustre: server umount lustre-MDT0001 complete [ 6129.524670] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 6132.027633] Key type lgssc unregistered [ 6132.458251] LNet: 189986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6132.471125] LNetError: 189986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6132.529669] LNet: Removed LNI 192.168.202.158@tcp [ 6133.894496] Key type .llcrypt unregistered [ 6133.896502] Key type ._llcrypt unregistered [ 6167.281552] Key type ._llcrypt registered [ 6167.282711] Key type .llcrypt registered [ 6167.462201] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 6169.478057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6169.553737] alg: No test for adler32 (adler32-zlib) [ 6171.342304] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6171.806860] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 6173.727483] Key type lgssc registered [ 6176.113643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6184.993413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6185.014309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.390454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6191.296815] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6196.717394] LustreError: 191404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6196.738976] LustreError: 191404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6200.649076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6201.096741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6206.228253] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6213.649963] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6219.242078] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6226.324045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6226.722261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6231.877273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6232.183939] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6240.557829] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6245.827505] Lustre: Failing over lustre-MDT0000 [ 6246.148315] Lustre: server umount lustre-MDT0000 complete [ 6247.405391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6247.421885] LustreError: 193246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.425833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6247.461199] LustreError: 193246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6250.464360] LustreError: 191403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6250.476825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6250.518083] Lustre: Skipped 1 previous similar message [ 6252.515115] LustreError: 191399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6252.523779] LustreError: 191399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6257.613063] LustreError: 191398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6257.637338] LustreError: 191398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6265.836277] LustreError: 193246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.866070] LustreError: 193246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6266.620562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.742468] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6266.987543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6267.078337] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6267.851057] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6272.021815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6272.080469] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6272.166622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6272.927985] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6278.540388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6282.700513] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6298.087406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6298.092428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.115916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.039993] Lustre: server umount lustre-OST0000 complete [ 6307.808718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6307.825085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6307.831615] Lustre: Skipped 1 previous similar message [ 6310.324609] Lustre: server umount lustre-MDT0000 complete [ 6312.928339] LustreError: 191400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.941473] LustreError: 191400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6313.939635] LustreError: 194983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448623 with bad export cookie 1096860368080060642 [ 6313.941038] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6313.946142] LustreError: 194983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6323.409549] Lustre: server umount lustre-MDT0001 complete [ 6337.510426] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 6341.707611] Key type lgssc unregistered [ 6342.345925] LNet: 196222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6342.356120] LNetError: 196222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6342.390228] LNet: Removed LNI 192.168.202.158@tcp [ 6344.046989] Key type .llcrypt unregistered [ 6344.051222] Key type ._llcrypt unregistered [ 6372.959530] Key type ._llcrypt registered [ 6372.961364] Key type .llcrypt registered [ 6373.104311] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 6374.656180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6374.723775] alg: No test for adler32 (adler32-zlib) [ 6375.780862] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6376.038506] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 6377.743264] Key type lgssc registered [ 6379.307934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6387.826235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6387.838405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6389.276300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6393.924708] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6399.464918] LustreError: 197635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.473453] LustreError: 197635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6403.781531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6404.160090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6409.202552] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6417.644793] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6423.609573] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6430.523475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6431.046042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6434.178767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6438.529923] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6446.239243] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6449.359352] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6453.947272] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6467.043254] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6475.240722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.242592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6475.244731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6475.247779] Lustre: Skipped 1 previous similar message [ 6475.270558] LustreError: Skipped 1 previous similar message [ 6480.355337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6480.358066] Lustre: Skipped 1 previous similar message [ 6480.565692] Lustre: server umount lustre-OST0000 complete [ 6485.478219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6485.501745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6485.508762] Lustre: Skipped 2 previous similar messages [ 6490.592777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6490.597939] Lustre: Skipped 1 previous similar message [ 6495.715863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6495.725316] Lustre: Skipped 1 previous similar message [ 6498.271321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6498.525323] Lustre: server umount lustre-MDT0000 complete [ 6500.835061] LustreError: 197635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6500.850747] LustreError: 197635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6502.773170] LustreError: 197615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448812 with bad export cookie 17168584737099439411 [ 6502.776713] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6502.782174] LustreError: 197615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6503.510443] Lustre: server umount lustre-MDT0001 complete [ 6517.496528] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 6519.916421] Key type lgssc unregistered [ 6520.121858] LNet: 201487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6520.125708] LNetError: 201487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6521.190508] LNet: Removed LNI 192.168.202.158@tcp [ 6522.221614] Key type .llcrypt unregistered [ 6522.223430] Key type ._llcrypt unregistered [ 6541.440651] Key type ._llcrypt registered [ 6541.442635] Key type .llcrypt registered [ 6541.641275] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 6542.739959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6542.788143] alg: No test for adler32 (adler32-zlib) [ 6543.931899] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6544.200712] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 6545.911289] Key type lgssc registered [ 6546.693262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6555.869792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6555.885716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6557.599779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6562.805981] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6567.906444] LustreError: 202753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.966325] LustreError: 202753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6573.033369] LustreError: 202754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6574.316045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6574.809206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6580.383248] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6588.347717] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6594.052524] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6600.069089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6600.549438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6602.622498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6607.468828] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6614.540733] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6619.896903] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6630.042968] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6638.041434] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6645.448696] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6653.921296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6653.925928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6653.938026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6655.969260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6655.975188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6655.981146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6656.011942] Lustre: Skipped 1 previous similar message [ 6661.090064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6666.209167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6666.221356] Lustre: Skipped 2 previous similar messages [ 6667.231455] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6667.406638] Lustre: server umount lustre-OST0000 complete [ 6671.329517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6671.337487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6671.341613] Lustre: Skipped 1 previous similar message [ 6671.358702] Lustre: Skipped 1 previous similar message [ 6677.297440] Lustre: server umount lustre-MDT0000 complete [ 6680.881475] LustreError: 202734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448990 with bad export cookie 4277048621319356036 [ 6680.897684] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6681.154768] Lustre: server umount lustre-MDT0001 complete [ 6690.659255] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 6693.181886] Key type lgssc unregistered [ 6693.420138] LNet: 207585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6693.423853] LNetError: 207585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6694.440276] LNet: Removed LNI 192.168.202.158@tcp [ 6695.433825] Key type .llcrypt unregistered [ 6695.435656] Key type ._llcrypt unregistered [ 6717.662916] Key type ._llcrypt registered [ 6717.666875] Key type .llcrypt registered [ 6717.747954] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 6718.639231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6718.765687] alg: No test for adler32 (adler32-zlib) [ 6719.803743] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6720.229045] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 6722.000675] Key type lgssc registered [ 6723.425359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6731.812197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6731.837609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6733.309194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6737.735344] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6743.526411] LustreError: 208991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6743.537400] LustreError: 208991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6746.456190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.987809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6751.298806] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6757.807703] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6763.612266] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6769.329438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6769.689754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6775.701587] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6778.928193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6783.802534] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6799.353111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6799.353648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6799.362270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6799.383634] Lustre: Skipped 1 previous similar message [ 6804.331200] Lustre: server umount lustre-OST0000 complete [ 6809.571564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6809.599103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6809.601908] Lustre: Skipped 1 previous similar message [ 6814.690128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6814.698850] Lustre: Skipped 1 previous similar message [ 6819.812042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6819.832753] Lustre: Skipped 1 previous similar message [ 6823.391093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6823.980272] Lustre: server umount lustre-MDT0000 complete [ 6824.933514] LustreError: 208991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6824.967615] LustreError: 208991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6829.316559] LustreError: 208973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449139 with bad export cookie 9088718570266975346 [ 6829.339305] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6829.343357] LustreError: 208973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6829.676131] Lustre: server umount lustre-MDT0001 complete [ 6842.212335] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 6845.708573] Key type lgssc unregistered [ 6846.116241] LNet: 212681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6846.132200] LNetError: 212681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6846.161367] LNet: Removed LNI 192.168.202.158@tcp [ 6847.315250] Key type .llcrypt unregistered [ 6847.321561] Key type ._llcrypt unregistered [ 6867.440351] Key type ._llcrypt registered [ 6867.445961] Key type .llcrypt registered [ 6867.621236] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 6868.827455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6868.952643] alg: No test for adler32 (adler32-zlib) [ 6870.210852] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6870.541989] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 6872.271184] Key type lgssc registered [ 6873.445305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6881.694747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6881.704171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.338919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6888.362996] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6893.544314] LustreError: 213947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6893.561146] LustreError: 213947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6897.897641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.203092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6903.435387] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6910.880775] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6915.701757] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6922.088625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6922.646345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6928.833442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6929.349971] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6939.279825] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6954.470246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6954.475775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6954.488924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6954.498340] Lustre: Skipped 1 previous similar message [ 6957.984983] Lustre: server umount lustre-OST0000 complete [ 6964.705622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6964.705679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6964.720231] Lustre: Skipped 1 previous similar message [ 6964.722070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6964.726035] LustreError: Skipped 1 previous similar message [ 6964.746978] Lustre: Skipped 1 previous similar message [ 6967.771331] Lustre: server umount lustre-MDT0000 complete [ 6969.827415] LustreError: 213947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6969.857245] LustreError: 213947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6971.820445] LustreError: 213927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449281 with bad export cookie 8860612760405046493 [ 6971.830468] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6971.831448] LustreError: 213927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6972.103603] Lustre: server umount lustre-MDT0001 complete [ 6982.734527] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 6985.934982] Key type lgssc unregistered [ 6986.372724] LNet: 217632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6986.396592] LNetError: 217632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6986.421146] LNet: Removed LNI 192.168.202.158@tcp [ 6987.559992] Key type .llcrypt unregistered [ 6987.562625] Key type ._llcrypt unregistered [ 7007.962295] Key type ._llcrypt registered [ 7007.963950] Key type .llcrypt registered [ 7008.047616] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 7008.978124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7008.989983] alg: No test for adler32 (adler32-zlib) [ 7010.211525] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7010.392719] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 7012.047177] Key type lgssc registered [ 7013.069355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7021.023913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7021.046149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7022.568902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7027.659555] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7032.801181] LustreError: 218898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7032.819837] LustreError: 218898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7036.863418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7037.267838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7042.679298] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7050.859659] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7057.018935] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7064.334306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7064.814528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7067.911134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7071.037189] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7078.206917] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7086.978236] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7087.020643] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7087.211284] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7093.669105] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7102.536198] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7104.894387] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7117.384874] Lustre: Permanently deactivating lustre-OST0001 [ 7117.413814] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7124.087074] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7131.905929] Lustre: Permanently reactivating lustre-OST0001 [ 7131.944773] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7131.949631] Lustre: Skipped 2 previous similar messages [ 7132.106342] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7132.124810] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7132.134628] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7132.167341] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7132.221507] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7147.489318] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7147.498194] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7147.508653] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7147.510562] Lustre: Skipped 2 previous similar messages [ 7148.522085] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7148.548113] Lustre: Skipped 2 previous similar messages [ 7153.655264] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7153.667681] Lustre: Skipped 1 previous similar message [ 7158.239894] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7158.328667] Lustre: server umount lustre-OST0001 complete [ 7158.755972] LustreError: 222225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7158.786253] LustreError: 222225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7163.889828] LustreError: 220719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7163.891856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7163.912237] LustreError: 220719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7163.916282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7163.928692] Lustre: Skipped 1 previous similar message [ 7163.995060] Lustre: Skipped 1 previous similar message [ 7168.998183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7169.000215] LustreError: 220720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7169.022978] Lustre: Skipped 1 previous similar message [ 7169.052076] LustreError: 220720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7174.144083] LustreError: 222225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7174.162056] LustreError: 222225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7177.183153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7177.397668] Lustre: server umount lustre-OST0000 complete [ 7184.353354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7184.356208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7184.384808] Lustre: Skipped 1 previous similar message [ 7184.402163] Lustre: Skipped 3 previous similar messages [ 7188.507247] Lustre: server umount lustre-MDT0000 complete [ 7189.475990] LustreError: 218898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7189.499398] LustreError: 218898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7192.069614] LustreError: 218878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449502 with bad export cookie 274673024498544639 [ 7192.076659] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7192.088327] LustreError: 218878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7192.371477] Lustre: server umount lustre-MDT0001 complete [ 7206.794495] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 7209.718572] Key type lgssc unregistered [ 7210.032591] LNet: 223956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7210.063754] LNetError: 223956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7210.093187] LNet: Removed LNI 192.168.202.158@tcp [ 7211.141899] Key type .llcrypt unregistered [ 7211.144407] Key type ._llcrypt unregistered [ 7257.817305] Key type ._llcrypt registered [ 7257.819771] Key type .llcrypt registered [ 7257.984537] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 7259.417644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7259.447387] alg: No test for adler32 (adler32-zlib) [ 7260.689061] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7261.086312] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 7262.776439] Key type lgssc registered [ 7264.208619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7274.033143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7274.055109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7275.405114] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7275.445239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7275.705407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7281.597547] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7291.139844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7291.258628] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7291.306354] Lustre: 226165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7291.691218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7296.151545] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7305.217759] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7311.411226] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7318.725744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7318.852582] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7319.034584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7325.642917] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7327.892744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7334.223400] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7337.770226] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7341.221224] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7349.130198] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7359.351224] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7367.752503] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7386.348890] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7386.363331] Lustre: Skipped 2 previous similar messages [ 7420.575477] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7420.578982] Lustre: Skipped 3 previous similar messages [ 7488.078832] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7488.083524] Lustre: Skipped 7 previous similar messages [ 7512.547845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7512.553570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7512.572120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7517.162407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7517.180067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7517.205987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7518.687691] Lustre: server umount lustre-OST0000 complete [ 7522.786162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7522.802493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7522.805138] Lustre: Skipped 1 previous similar message [ 7527.403624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7527.414245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7527.434115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7528.889782] Lustre: server umount lustre-MDT0000 complete [ 7532.554393] LustreError: 228036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449842 with bad export cookie 2064113720177216777 [ 7532.557527] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7532.562878] LustreError: 228036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7532.848992] Lustre: server umount lustre-MDT0001 complete [ 7544.164678] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 7547.456947] Key type lgssc unregistered [ 7547.844555] LNet: 230984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7547.849835] LNetError: 230984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7547.880773] LNet: Removed LNI 192.168.202.158@tcp [ 7548.800087] Key type .llcrypt unregistered [ 7548.803505] Key type ._llcrypt unregistered [ 7573.583960] Key type ._llcrypt registered [ 7573.586155] Key type .llcrypt registered [ 7573.723730] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 7575.436314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7575.593027] alg: No test for adler32 (adler32-zlib) [ 7576.907257] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7577.185991] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 7578.839219] Key type lgssc registered [ 7580.380444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7591.076445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7591.105715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7592.664993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7598.143522] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7602.210044] LustreError: 232400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7602.228814] LustreError: 232400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7607.284814] LustreError: 232401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7608.320801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.104827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7615.310795] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7623.606169] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7631.340591] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7638.854852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7639.311652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.709418] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7655.098984] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7663.935827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7679.457102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7679.466169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7679.489407] Lustre: Skipped 1 previous similar message [ 7679.490624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7681.057918] Lustre: server umount lustre-OST0000 complete [ 7686.112633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7686.126161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7686.141340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7686.154211] Lustre: Skipped 1 previous similar message [ 7689.704517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7689.705421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7691.287431] Lustre: server umount lustre-MDT0000 complete [ 7694.817268] LustreError: 232400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7694.837306] LustreError: 232400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7695.351239] LustreError: 232380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450005 with bad export cookie 11520427333361238904 [ 7695.353438] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7695.367778] LustreError: 232380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7695.899255] Lustre: server umount lustre-MDT0001 complete [ 7708.528250] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 7711.447785] Key type lgssc unregistered [ 7711.684844] LNet: 236115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7711.695782] LNetError: 236115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7711.717880] LNet: Removed LNI 192.168.202.158@tcp [ 7712.843269] Key type .llcrypt unregistered [ 7712.845828] Key type ._llcrypt unregistered [ 7735.051523] Key type ._llcrypt registered [ 7735.053476] Key type .llcrypt registered [ 7735.177817] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 7736.424414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7736.615921] alg: No test for adler32 (adler32-zlib) [ 7737.741440] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7737.973286] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 7739.647374] Key type lgssc registered [ 7740.956388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8010.685422] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 8013.510251] Key type lgssc unregistered [ 8013.776761] LNet: 238141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8013.779893] LNetError: 238141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8013.789396] LNet: Removed LNI 192.168.202.158@tcp [ 8014.572401] Key type .llcrypt unregistered [ 8014.573647] Key type ._llcrypt unregistered [ 8041.508262] Key type ._llcrypt registered [ 8041.509749] Key type .llcrypt registered [ 8041.580623] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_hostid [ 8056.124143] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 8056.800582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8056.826313] alg: No test for adler32 (adler32-zlib) [ 8057.822723] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8058.079883] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 8059.711199] Key type lgssc registered [ 8060.586112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8087.169830] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7776 sec ========== 11:46:36 (1788450396) [ 8088.535865] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:46:37 (1788450397) === [ 8091.474872] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:46:40 (1788450400) === [ 8128.827639] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 8131.384277] Key type lgssc unregistered [ 8131.642168] LNet: 243377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8131.661290] LNetError: 243377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8131.675158] LNet: Removed LNI 192.168.202.158@tcp [ 8132.453759] Key type .llcrypt unregistered [ 8132.455384] Key type ._llcrypt unregistered