[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 515545207 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2480MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002329] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007519] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011141] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.022852] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028402] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032084] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033115] *** VALIDATE tmpfs *** [ 0.034514] *** VALIDATE proc *** [ 0.036128] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.039148] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042026] Spectre V2 : User space: Vulnerable [ 0.043006] Speculative Store Bypass: Vulnerable [ 0.045982] debug: unmapping init [mem 0xffffffffa7e59000-0xffffffffa7e60fff] [ 0.047160] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048521] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049019] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051012] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053012] ... max period: 00007fffffffffff [ 0.054014] ... fixed-purpose events: 3 [ 0.054865] ... event mask: 000000070000000f [ 0.055290] rcu: Hierarchical SRCU implementation. [ 0.057402] smp: Bringing up secondary CPUs ... [ 0.058580] x86: Booting SMP configuration: [ 0.059023] .... node #0, CPUs: #1 #2 #3 [ 0.062259] smp: Brought up 1 node, 4 CPUs [ 0.064009] smpboot: Max logical packages: 1 [ 0.064993] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141064] node 0 deferred pages initialised in 75ms [ 0.144008] devtmpfs: initialized [ 0.145280] x86/mm: Memory block size: 128MB [ 0.148048] gcov: version magic: 0x41383552 [ 0.150291] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152320] pinctrl core: initialized pinctrl subsystem [ 0.153258] [ 0.153773] ************************************************************* [ 0.154017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155013] ** ** [ 0.156015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157018] ** ** [ 0.158013] ** This means that this kernel is built to expose internal ** [ 0.159013] ** IOMMU data structures, which may compromise security on ** [ 0.160013] ** your system. ** [ 0.161017] ** ** [ 0.162012] ** If you see this message and you are not debugging the ** [ 0.163020] ** kernel, report this immediately to your vendor! ** [ 0.164016] ** ** [ 0.165015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166016] ************************************************************* [ 0.167881] NET: Registered protocol family 16 [ 0.168538] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.171496] cpuidle: using governor menu [ 0.172000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176652] PCI: Using configuration type 1 for base access [ 0.178215] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189038] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192070] cryptd: max_cpu_qlen set to 1000 [ 0.194109] ACPI: Added _OSI(Module Device) [ 0.195013] ACPI: Added _OSI(Processor Device) [ 0.196014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.197023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201729] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.204134] ACPI: Interpreter enabled [ 0.205056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.206012] ACPI: Using IOAPIC for interrupt routing [ 0.207114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.208409] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.217929] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.219020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.220094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.222583] acpiphp: Slot [2] registered [ 0.223125] acpiphp: Slot [5] registered [ 0.224102] acpiphp: Slot [6] registered [ 0.225094] acpiphp: Slot [7] registered [ 0.226109] acpiphp: Slot [8] registered [ 0.227101] acpiphp: Slot [9] registered [ 0.228088] acpiphp: Slot [10] registered [ 0.229119] acpiphp: Slot [3] registered [ 0.230079] acpiphp: Slot [4] registered [ 0.231093] acpiphp: Slot [11] registered [ 0.232086] acpiphp: Slot [12] registered [ 0.233072] acpiphp: Slot [13] registered [ 0.234080] acpiphp: Slot [14] registered [ 0.235077] acpiphp: Slot [15] registered [ 0.236105] acpiphp: Slot [16] registered [ 0.237100] acpiphp: Slot [17] registered [ 0.238085] acpiphp: Slot [18] registered [ 0.239104] acpiphp: Slot [19] registered [ 0.240092] acpiphp: Slot [20] registered [ 0.241087] acpiphp: Slot [21] registered [ 0.242106] acpiphp: Slot [22] registered [ 0.243088] acpiphp: Slot [23] registered [ 0.244088] acpiphp: Slot [24] registered [ 0.245091] acpiphp: Slot [25] registered [ 0.246098] acpiphp: Slot [26] registered [ 0.247108] acpiphp: Slot [27] registered [ 0.248105] acpiphp: Slot [28] registered [ 0.249099] acpiphp: Slot [29] registered [ 0.250112] acpiphp: Slot [30] registered [ 0.251088] acpiphp: Slot [31] registered [ 0.252051] PCI host bridge to bus 0000:00 [ 0.253018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.254018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.255019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.256030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.257020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.258023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.259172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.260966] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.262202] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.267012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.270063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.271024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.272020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.273032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.274625] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.275794] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.276047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.277833] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.280017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.290022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.292019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.296025] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.305019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.312019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.332019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.343426] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.354016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.368015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.389018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.413222] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.430021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.437021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.464019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.478327] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.495018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.502020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.523017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.537092] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.551016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.572022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.589019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.599439] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.607019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.618019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.640015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.664092] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.667435] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.670480] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.672359] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.675248] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.679187] iommu: Default domain type: Passthrough [ 0.681566] SCSI subsystem initialized [ 0.683159] ACPI: bus type USB registered [ 0.685116] usbcore: registered new interface driver usbfs [ 0.687341] usbcore: registered new interface driver hub [ 0.690112] usbcore: registered new device driver usb [ 0.692155] pps_core: LinuxPPS API ver. 1 registered [ 0.694011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.697063] PTP clock support registered [ 0.699130] EDAC MC: Ver: 3.0.0 [ 0.701109] PCI: Using ACPI for IRQ routing [ 0.703940] NetLabel: Initializing [ 0.705016] NetLabel: domain hash size = 128 [ 0.706009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.708112] NetLabel: unlabeled traffic allowed by default [ 0.711077] vgaarb: loaded [ 0.712175] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.714015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.722014] clocksource: Switched to clocksource kvm-clock [ 0.823159] VFS: Disk quotas dquot_6.6.0 [ 0.824882] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.827235] *** VALIDATE ramfs *** [ 0.828364] *** VALIDATE hugetlbfs *** [ 0.829740] pnp: PnP ACPI init [ 0.832062] pnp: PnP ACPI: found 6 devices [ 0.849056] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.851987] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.853893] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.855799] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.857970] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.860287] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.862606] NET: Registered protocol family 2 [ 0.864541] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.869253] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.872775] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.877832] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.881166] TCP: Hash tables configured (established 65536 bind 65536) [ 0.884180] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.887279] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.889920] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.893225] NET: Registered protocol family 1 [ 0.895887] RPC: Registered named UNIX socket transport module. [ 0.898169] RPC: Registered udp transport module. [ 0.900033] RPC: Registered tcp transport module. [ 0.902088] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.904442] NET: Registered protocol family 44 [ 0.905829] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.907268] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.909150] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.911594] PCI: CLS 0 bytes, default 64 [ 0.913336] Unpacking initramfs... [ 2.297556] debug: unmapping init [mem 0xffff917dfcc54000-0xffff917dfffbffff] [ 2.299985] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.301323] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.303107] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.812502] Initialise system trusted keyrings [ 2.814307] Key type blacklist registered [ 2.816450] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.826284] zbud: loaded [ 2.829025] *** VALIDATE nfs *** [ 2.830446] *** VALIDATE nfs4 *** [ 2.831945] pstore: using deflate compression [ 2.836248] Platform Keyring initialized [ 2.947285] NET: Registered protocol family 38 [ 2.948935] Key type asymmetric registered [ 2.950062] Asymmetric key parser 'x509' registered [ 2.951361] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.953660] io scheduler mq-deadline registered [ 2.956587] io scheduler kyber registered [ 2.957590] io scheduler bfq registered [ 2.959108] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.961492] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.963813] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.966210] ACPI: Power Button [PWRF] [ 2.971863] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.978347] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.988974] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.994808] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.007968] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.036113] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.065053] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.070179] Non-volatile memory driver v1.3 [ 3.071916] Linux agpgart interface v0.103 [ 3.103303] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.106196] vda: detected capacity change from 0 to 75083776 [ 3.121424] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.124119] vdb: detected capacity change from 0 to 1073741824 [ 3.139385] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.142475] vdc: detected capacity change from 0 to 2621440000 [ 3.159811] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.163482] vdd: detected capacity change from 0 to 2621440000 [ 3.176933] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.179816] vde: detected capacity change from 0 to 4294967296 [ 3.201259] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.203575] vdf: detected capacity change from 0 to 4294967296 [ 3.211903] libphy: Fixed MDIO Bus: probed [ 3.225971] usbcore: registered new interface driver usbserial_generic [ 3.227404] usbserial: USB Serial support registered for generic [ 3.228625] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.231304] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.232451] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.234040] mousedev: PS/2 mouse device common for all mice [ 3.236801] rtc_cmos 00:05: RTC can wake from S4 [ 3.238855] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.239670] rtc_cmos 00:05: registered as rtc0 [ 3.243270] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.244634] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.245430] intel_pstate: CPU model not supported [ 3.249698] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.251262] hid: raw HID events driver (C) Jiri Kosina [ 3.254162] usbcore: registered new interface driver usbhid [ 3.255497] usbhid: USB HID core driver [ 3.256399] drop_monitor: Initializing network drop monitor service [ 3.257750] Initializing XFRM netlink socket [ 3.258984] NET: Registered protocol family 10 [ 3.260946] Segment Routing with IPv6 [ 3.262428] NET: Registered protocol family 17 [ 3.264608] mpls_gso: MPLS GSO support [ 3.269582] RAS: Correctable Errors collector initialized. [ 3.271692] AVX version of gcm_enc/dec engaged. [ 3.272959] AES CTR mode by8 optimization enabled [ 3.343663] sched_clock: Marking stable (3343644633, 0)->(4304248700, -960604067) [ 3.347537] registered taskstats version 1 [ 3.349327] Loading compiled-in X.509 certificates [ 3.350896] zswap: loaded using pool lzo/zbud [ 3.371326] Key type big_key registered [ 3.381853] Key type encrypted registered [ 3.382922] ima: No TPM chip found, activating TPM-bypass! [ 3.384244] ima: Allocated hash algorithm: sha1 [ 3.385277] ima: No architecture policies found [ 3.386375] evm: Initialising EVM extended attributes: [ 3.387729] evm: security.selinux [ 3.388686] evm: security.ima [ 3.389433] evm: security.capability [ 3.390386] evm: HMAC attrs: 0x1 [ 3.392584] rtc_cmos 00:05: setting system clock to 2026-09-07 09:55:55 UTC (1788774955) [ 3.397310] debug: unmapping init [mem 0xffffffffa8e03000-0xffffffffa8ffffff] [ 3.400722] debug: unmapping init [mem 0xffffffffa7b82000-0xffffffffa7e58fff] [ 3.409163] Write protecting the kernel read-only data: 28672k [ 3.412779] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 3.414681] debug: unmapping init [mem 0xffffffffa6b14000-0xffffffffa6bfffff] [ 3.444952] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.452847] systemd[1]: Detected virtualization kvm. [ 3.455088] systemd[1]: Detected architecture x86-64. [ 3.457045] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.485567] systemd[1]: No hostname configured. [ 3.487465] systemd[1]: Set hostname to . [ 3.489414] random: systemd: uninitialized urandom read (16 bytes read) [ 3.491934] systemd[1]: Initializing machine ID from random generator. [ 3.620973] random: systemd: uninitialized urandom read (16 bytes read) [ 3.623581] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.628125] random: systemd: uninitialized urandom read (16 bytes read) [ 3.630764] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.634438] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.175915] device-mapper: uevent: version 1.0.3 [ 4.178336] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.878562] random: fast init done [ 4.906864] virtio_net virtio0 ens2: renamed from eth0 [ 5.020514] scsi host0: ata_piix [ 5.053484] scsi host1: ata_piix [ 5.054797] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.056847] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.540916] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.911132] random: crng init done [ 9.912063] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.267702] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.539643] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.825346] SELinux: Disabled at runtime. [ 11.885243] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.893717] systemd[1]: Detected virtualization kvm. [ 11.895584] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.464257] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.467940] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.473043] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.479362] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.483105] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.490356] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.501277] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [[ 12.711189] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 14.394708] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.912780] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.961502] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.520269] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.630286] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 21.334926] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 22.283387] NFS: Registering the id_resolver key type [ 22.299246] Key type id_resolver registered [ 22.305604] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg213-server login: [ 80.759008] hrtimer: interrupt took 5000615 ns [ 93.167776] libcfs: loading out-of-tree module taints kernel. [ 93.213228] Key type ._llcrypt registered [ 93.218054] Key type .llcrypt registered [ 93.419515] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 115.826319] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 118.245499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 118.281591] alg: No test for adler32 (adler32-zlib) [ 120.129602] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 121.464855] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 123.359153] Key type lgssc registered [ 125.849025] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.726366] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 200.291287] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 218.165580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 218.216861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.583455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 219.609448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 219.712309] Lustre: lustre-MDT0000: new disk, initializing [ 219.891862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.942281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 226.293410] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.308778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 243.435949] Lustre: 6528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 243.486230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 243.490873] Lustre: Skipped 1 previous similar message [ 243.630407] Lustre: lustre-MDT0001: new disk, initializing [ 243.742733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.797957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 243.825977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 249.347284] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.073683] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 267.497955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.810190] Lustre: lustre-OST0000: new disk, initializing [ 267.812798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 267.818535] Lustre: 8467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 267.907489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 275.484180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 275.488465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 275.586572] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.596791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 292.653300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 292.769305] Lustre: lustre-OST0001: new disk, initializing [ 292.774528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 292.779728] Lustre: 9540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 292.885709] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 300.975147] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 301.117175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 301.133778] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 301.217751] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 315.011430] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 326.144542] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 333.271682] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing check_logdir /tmp/testlogs/ [ 341.798584] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing yml_node [ 346.836929] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 350.332263] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 353.446790] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 355.955160] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 06:01:46 EDT 2026 [ 377.225577] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 379.438257] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 398.304848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.308667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 398.340877] Lustre: Skipped 2 previous similar messages [ 398.357600] Lustre: Skipped 3 previous similar messages [ 403.430436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 403.452430] Lustre: Skipped 3 previous similar messages [ 404.686292] Lustre: server umount lustre-MDT0000 complete [ 409.839221] LustreError: 10369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788775361 with bad export cookie 11433955076608525773 [ 409.844824] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 409.850085] LustreError: 10369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 413.671134] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 413.674960] LustreError: 12463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 413.675111] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 413.695393] Lustre: Skipped 2 previous similar messages [ 413.742353] LustreError: 12463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 414.688493] LustreError: 6535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 414.706546] LustreError: 6535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 416.470475] Lustre: server umount lustre-MDT0001 complete [ 422.910591] Lustre: server umount lustre-OST0000 complete [ 428.140256] Lustre: server umount lustre-OST0001 complete [ 437.307083] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 446.702784] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 492.402567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.586217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 492.626765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 492.720902] Lustre: lustre-MDT0000: new disk, initializing [ 492.815922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.833522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 498.689582] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 510.279457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.351970] Lustre: 15116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 510.415911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 510.418466] Lustre: Skipped 1 previous similar message [ 510.535971] Lustre: lustre-MDT0001: new disk, initializing [ 510.706682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 510.829254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 510.848784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 516.390542] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 526.713080] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 533.261750] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 541.530040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 541.831591] Lustre: lustre-OST0000: new disk, initializing [ 541.840879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 541.853984] Lustre: 17085:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 541.964483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.710578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 543.735873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 543.887178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 549.385119] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 560.781859] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 566.211391] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 566.493283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 571.953737] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 572.300149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 574.944868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 574.947993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 574.968265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 574.970267] Lustre: Skipped 3 previous similar messages [ 579.553037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.556070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 579.579876] Lustre: Skipped 1 previous similar message [ 580.907961] Lustre: server umount lustre-OST0000 complete [ 587.748934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 587.758490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.781648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 591.261652] Lustre: server umount lustre-MDT0000 complete [ 594.914057] LustreError: 15126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.935965] LustreError: 15126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 595.397906] LustreError: 15107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788775547 with bad export cookie 11433955076608527313 [ 595.413833] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.415812] LustreError: 15107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 596.039770] Lustre: server umount lustre-MDT0001 complete [ 600.831619] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 06:05:51 (1788775551) [ 610.651765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.546527] LustreError: 19352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.620479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 616.930842] LustreError: 19353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.328406] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 626.146507] LustreError: 19352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.190842] LustreError: 19352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 628.972505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.761940] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 643.548748] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 649.863503] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 657.638154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 665.885215] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 674.164917] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 695.265576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 695.274213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.309597] Lustre: Skipped 1 previous similar message [ 695.315474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 695.330838] Lustre: Skipped 2 previous similar messages [ 698.357388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 699.780356] Lustre: server umount lustre-OST0000 complete [ 706.533148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 718.303392] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 718.623056] Lustre: server umount lustre-MDT0000 complete [ 718.821741] LustreError: 19353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.850561] LustreError: 19353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 723.283039] LustreError: 19334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788775675 with bad export cookie 11433955076608528195 [ 723.284748] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.292016] LustreError: 19334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 723.790535] Lustre: server umount lustre-MDT0001 complete [ 738.127476] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 741.578915] Key type lgssc unregistered [ 741.896930] LNet: 22916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.910958] LNetError: 22916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.936924] LNet: Removed LNI 192.168.202.113@tcp [ 742.873171] Key type .llcrypt unregistered [ 742.876573] Key type ._llcrypt unregistered [ 773.918581] Key type ._llcrypt registered [ 773.920519] Key type .llcrypt registered [ 774.032837] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 775.879944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 776.040459] alg: No test for adler32 (adler32-zlib) [ 777.281871] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 777.611357] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 779.280892] Key type lgssc registered [ 781.153926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 790.894028] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 790.924834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 792.522537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 796.646341] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 802.787670] LustreError: 24325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 802.827486] LustreError: 24325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 807.527271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.969639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 812.805399] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 820.081883] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 826.692231] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 835.448198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 836.018626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 837.045651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 843.460295] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 852.519584] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 867.812653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 867.813605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 867.816873] LustreError: Skipped 1 previous similar message [ 867.818362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 867.824245] Lustre: Skipped 1 previous similar message [ 867.832154] Lustre: Skipped 1 previous similar message [ 869.643328] Lustre: server umount lustre-OST0000 complete [ 874.465865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 874.478471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.508896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.052200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.053553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.650626] Lustre: server umount lustre-MDT0000 complete [ 883.169916] LustreError: 26237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 883.204616] LustreError: 26237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 885.166332] LustreError: 24306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788775837 with bad export cookie 11114410937361668072 [ 885.176605] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 885.188899] LustreError: 24306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 885.904902] Lustre: server umount lustre-MDT0001 complete [ 899.490471] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 901.832717] Key type lgssc unregistered [ 902.313700] LNet: 28299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.327193] LNetError: 28299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.350584] LNet: Removed LNI 192.168.202.113@tcp [ 903.263895] Key type .llcrypt unregistered [ 903.265534] Key type ._llcrypt unregistered [ 929.585739] Key type ._llcrypt registered [ 929.587935] Key type .llcrypt registered [ 929.692467] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 930.773176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 930.951547] alg: No test for adler32 (adler32-zlib) [ 932.103575] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 932.328114] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 933.975259] Key type lgssc registered [ 935.127721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 944.198249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 944.213450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 945.818087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.167936] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 955.425962] LustreError: 29713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 955.440948] LustreError: 29713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 960.492476] LustreError: 29712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 961.366258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.774035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 966.597758] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 976.272379] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 982.703385] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 997.095641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 997.394032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.383631] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1012.220212] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1015.866042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1026.019163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.026130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1026.032339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1031.139645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1031.154591] Lustre: Skipped 1 previous similar message [ 1031.506749] Lustre: server umount lustre-OST0000 complete [ 1036.259732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.261811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1036.282770] Lustre: Skipped 2 previous similar messages [ 1041.376344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1041.386054] Lustre: Skipped 2 previous similar messages [ 1042.390527] Lustre: server umount lustre-MDT0000 complete [ 1046.253791] LustreError: 30448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788775998 with bad export cookie 10633491721938444405 [ 1046.264531] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1046.276081] LustreError: 30448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1046.821773] Lustre: server umount lustre-MDT0001 complete [ 1059.250700] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1062.346434] Key type lgssc unregistered [ 1062.753904] LNet: 33614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.775754] LNetError: 33614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.850177] LNet: Removed LNI 192.168.202.113@tcp [ 1065.013396] Key type .llcrypt unregistered [ 1065.015843] Key type ._llcrypt unregistered [ 1091.404047] Key type ._llcrypt registered [ 1091.405670] Key type .llcrypt registered [ 1091.641892] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1094.361218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1094.479108] alg: No test for adler32 (adler32-zlib) [ 1095.733901] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1096.019331] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1097.727462] Key type lgssc registered [ 1099.353741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1106.642802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1106.663324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.167820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.103802] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1118.693327] LustreError: 35024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.729254] LustreError: 35024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1122.663083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.173982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1127.791415] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1135.601530] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1141.951574] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1148.963397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1149.540814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1153.643915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1156.406236] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1164.801193] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1178.088963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1178.096748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.105208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1179.616364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1179.623728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.649867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1182.704710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1184.657367] Lustre: server umount lustre-OST0000 complete [ 1189.860459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1189.878071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.893275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1189.897742] Lustre: Skipped 1 previous similar message [ 1192.934266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.050552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1198.056581] Lustre: Skipped 3 previous similar messages [ 1203.170027] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1203.400138] Lustre: server umount lustre-MDT0000 complete [ 1207.267670] LustreError: 35024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.293173] LustreError: 35024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1207.472344] LustreError: 35004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776159 with bad export cookie 3112216664297739595 [ 1207.487926] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1208.522411] Lustre: server umount lustre-MDT0001 complete [ 1221.852303] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1225.891480] Key type lgssc unregistered [ 1226.311273] LNet: 38589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.321481] LNetError: 38589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.374732] LNet: Removed LNI 192.168.202.113@tcp [ 1228.982968] Key type .llcrypt unregistered [ 1228.987977] Key type ._llcrypt unregistered [ 1258.427221] Key type ._llcrypt registered [ 1258.430320] Key type .llcrypt registered [ 1258.582988] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1259.941694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1260.054581] alg: No test for adler32 (adler32-zlib) [ 1261.537876] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1261.892129] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1263.655161] Key type lgssc registered [ 1265.571811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1275.484906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1275.523974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.021336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.406490] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1287.146501] LustreError: 40006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.181637] LustreError: 40006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1291.307393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.697435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1296.835161] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1305.038944] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1310.912790] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1318.113822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1318.480401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.636515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1325.065347] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1332.748984] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1343.465280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1343.471474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1343.482988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1345.909800] Lustre: lustre-OST0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 1346.528434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1346.540189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.988437] Lustre: lustre-OST0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 1351.002679] Lustre: Skipped 2 previous similar messages [ 1356.110401] Lustre: lustre-OST0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 1356.119127] Lustre: Skipped 2 previous similar messages [ 1357.279497] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1357.767076] Lustre: server umount lustre-OST0000 complete [ 1378.463191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.487230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1378.497065] Lustre: Skipped 3 previous similar messages [ 1385.072191] Lustre: server umount lustre-MDT0000 complete [ 1385.955618] LustreError: 40006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1385.981838] LustreError: 40006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1389.531219] LustreError: 39988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776341 with bad export cookie 12905082762959049490 [ 1389.547962] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1389.548971] LustreError: 39988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1396.011358] Lustre: server umount lustre-MDT0001 complete [ 1408.178975] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1411.203960] Key type lgssc unregistered [ 1411.562759] LNet: 43716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.587969] LNetError: 43716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.645814] LNet: Removed LNI 192.168.202.113@tcp [ 1413.777447] Key type .llcrypt unregistered [ 1413.783545] Key type ._llcrypt unregistered [ 1440.844191] Key type ._llcrypt registered [ 1440.845810] Key type .llcrypt registered [ 1441.003791] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1442.118528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1442.221649] alg: No test for adler32 (adler32-zlib) [ 1443.315181] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1443.585644] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1445.279199] Key type lgssc registered [ 1446.841255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1455.543433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1455.579033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1457.429733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.771124] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1467.872397] LustreError: 45125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.897977] LustreError: 45125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1470.979893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.417654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1475.894910] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1483.316683] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1489.685524] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1497.452521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1498.028344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1502.139710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1505.713196] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1514.114360] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1526.142830] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 1526.764428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.767491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1526.793695] Lustre: Skipped 1 previous similar message [ 1529.486879] Lustre: server umount lustre-MDT0000 complete [ 1531.212243] LustreError: 45119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1533.308363] LustreError: 45104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776485 with bad export cookie 16595631272367887041 [ 1533.310169] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1533.317325] LustreError: 45104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1536.333466] LustreError: 45119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1536.361546] LustreError: 45119:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1536.382971] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.13@tcp (stopping) [ 1536.388043] Lustre: Skipped 2 previous similar messages [ 1536.999772] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1537.008064] Lustre: Skipped 1 previous similar message [ 1539.040706] LustreError: 45121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1539.040889] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1539.071802] LustreError: 45121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1539.094048] Lustre: Skipped 1 previous similar message [ 1539.810539] Lustre: server umount lustre-MDT0001 complete [ 1555.447971] Lustre: server umount lustre-OST0000 complete [ 1574.468438] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1576.962317] Key type lgssc unregistered [ 1577.321942] LNet: 48986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.325333] LNetError: 48986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.335855] LNet: Removed LNI 192.168.202.113@tcp [ 1578.238124] Key type .llcrypt unregistered [ 1578.239900] Key type ._llcrypt unregistered [ 1605.653927] Key type ._llcrypt registered [ 1605.656948] Key type .llcrypt registered [ 1605.791643] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1606.899475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1606.936738] alg: No test for adler32 (adler32-zlib) [ 1608.155372] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1608.438912] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1610.072797] Key type lgssc registered [ 1611.338701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1619.495377] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1619.517132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1635.103596] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1640.223489] LustreError: 50370:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 1666.911720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.050528] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1682.132967] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1849.204462] Lustre: server umount lustre-OST0000 complete [ 1865.984479] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1869.010143] Key type lgssc unregistered [ 1869.427444] LNet: 52045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.439453] LNetError: 52045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.470123] LNet: Removed LNI 192.168.202.113@tcp [ 1870.555931] Key type .llcrypt unregistered [ 1870.559422] Key type ._llcrypt unregistered [ 1894.666631] Key type ._llcrypt registered [ 1894.673347] Key type .llcrypt registered [ 1894.866367] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1895.996374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1896.108531] alg: No test for adler32 (adler32-zlib) [ 1897.294486] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1897.629280] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1899.303172] Key type lgssc registered [ 1900.512913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1907.414154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1907.428864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.718457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.606256] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1917.928131] LustreError: 53456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1917.954328] LustreError: 53456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1920.659137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.067788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1925.248789] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1932.035195] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1937.887278] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1943.829701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1944.240514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.992729] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1953.465169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1957.943358] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1961.955527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1961.966878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.005567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1964.001572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.017783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1964.040921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1964.089628] Lustre: Skipped 1 previous similar message [ 1967.012536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1968.263201] Lustre: server umount lustre-OST0000 complete [ 1977.313080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1977.323831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.333298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1977.337504] Lustre: Skipped 1 previous similar message [ 1977.360725] Lustre: Skipped 2 previous similar messages [ 1978.738363] Lustre: server umount lustre-MDT0000 complete [ 1982.444431] LustreError: 55767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.481636] LustreError: 55767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1983.610688] LustreError: 53437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776935 with bad export cookie 10465741304694462601 [ 1983.629428] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.631542] LustreError: 53437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1984.139444] Lustre: server umount lustre-MDT0001 complete [ 1997.225088] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2000.937321] Key type lgssc unregistered [ 2001.213446] LNet: 56985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.220573] LNetError: 56985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.280964] LNet: Removed LNI 192.168.202.113@tcp [ 2003.083812] Key type .llcrypt unregistered [ 2003.086243] Key type ._llcrypt unregistered [ 2027.330842] Key type ._llcrypt registered [ 2027.333497] Key type .llcrypt registered [ 2027.449276] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2028.821835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2028.923440] alg: No test for adler32 (adler32-zlib) [ 2029.939639] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2030.173507] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2031.903185] Key type lgssc registered [ 2033.290376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2040.822759] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2040.836680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2056.353122] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2061.471519] LustreError: 58368:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 2088.159213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.953661] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2101.666453] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2107.450283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.883141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.936470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2112.700966] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2113.008440] LustreError: 58394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.032461] LustreError: 58394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2118.114722] LustreError: 58393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.476354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.876994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2126.818598] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2134.325359] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2140.699301] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2142.694996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2142.707964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.728597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2143.718061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2143.732944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2143.733970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.836686] LustreError: 58385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.860373] LustreError: 58385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2148.921801] Lustre: server umount lustre-OST0000 complete [ 2170.195824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.202842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.219056] Lustre: Skipped 1 previous similar message [ 2170.235527] Lustre: Skipped 1 previous similar message [ 2174.240730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2176.458624] Lustre: server umount lustre-MDT0000 complete [ 2179.361694] LustreError: 59410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.370946] LustreError: 59410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2180.130579] LustreError: 58374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777132 with bad export cookie 12004972652916518402 [ 2180.137904] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2180.142283] LustreError: 58374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2186.667416] Lustre: server umount lustre-MDT0001 complete [ 2198.372070] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2201.669921] Key type lgssc unregistered [ 2202.070669] LNet: 62109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.079404] LNetError: 62109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.107247] LNet: Removed LNI 192.168.202.113@tcp [ 2203.270678] Key type .llcrypt unregistered [ 2203.281356] Key type ._llcrypt unregistered [ 2228.457525] Key type ._llcrypt registered [ 2228.460187] Key type .llcrypt registered [ 2228.575697] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2229.663418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2229.991166] alg: No test for adler32 (adler32-zlib) [ 2231.061931] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2231.324982] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2233.039212] Key type lgssc registered [ 2234.095810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2241.364294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2241.381151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.668622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.183485] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2252.262868] LustreError: 63523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.279949] LustreError: 63523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2256.161613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.673052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2261.195233] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2268.384269] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2273.946994] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2279.906800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2280.355229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.005880] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2287.545166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2292.897991] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2312.161105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2312.171582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.194465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2313.185519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.187660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2314.882731] Lustre: server umount lustre-OST0000 complete [ 2323.425479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.426867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2323.431338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2323.431348] Lustre: Skipped 1 previous similar message [ 2325.184887] Lustre: server umount lustre-MDT0000 complete [ 2328.150660] LustreError: 63503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777280 with bad export cookie 3975829772432646858 [ 2328.175265] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2328.461132] Lustre: server umount lustre-MDT0001 complete [ 2341.575469] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2344.797912] Key type lgssc unregistered [ 2345.062661] LNet: 67075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.065692] LNetError: 67075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2345.075748] LNet: Removed LNI 192.168.202.113@tcp [ 2346.240857] Key type .llcrypt unregistered [ 2346.244126] Key type ._llcrypt unregistered [ 2374.027272] Key type ._llcrypt registered [ 2374.029618] Key type .llcrypt registered [ 2374.184458] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2375.262936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2375.353767] alg: No test for adler32 (adler32-zlib) [ 2376.460376] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2376.838821] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2378.511206] Key type lgssc registered [ 2379.494729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2388.686028] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 06:35:40 (1788777340) [ 2395.331378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2395.344479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.757690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.210512] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2406.888978] LustreError: 68774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.918161] LustreError: 68774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2410.571249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.174646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2416.006736] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2422.795528] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2428.160636] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2435.246910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2435.596262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.736147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2441.824637] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2449.124938] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2455.651074] Lustre: Failing over lustre-MDT0000 [ 2455.883842] Lustre: server umount lustre-MDT0000 complete [ 2456.952148] LustreError: 68769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.056715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2457.060156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.085930] Lustre: Skipped 2 previous similar messages [ 2462.030914] LustreError: 68771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.056576] LustreError: 68771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2464.797585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.974983] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2465.099556] Lustre: *** cfs_fail_loc=135, val=0*** [ 2465.192042] LustreError: 71862:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2465.196658] LustreError: 71862:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.113@tcp: cfg command failed: rc = -2 [ 2465.206088] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2465.206088] [ 2465.214633] LustreError: MGC192.168.202.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2465.227547] LustreError: 71851:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2465.232258] LustreError: 71851:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 2465.243699] LustreError: 71851:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2465.248949] LustreError: 71851:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2465.268926] Lustre: server umount lustre-MDT0000 complete [ 2465.272673] LustreError: 71851:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2467.159708] LustreError: 68769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2467.167963] LustreError: 68769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2472.219140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2472.271449] LustreError: 70673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.296685] LustreError: 70673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2472.358887] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2472.419842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2472.430012] Lustre: Skipped 2 previous similar messages [ 2472.548968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2472.594293] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2476.215551] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2477.387675] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2477.549172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2477.600180] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2477.737451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2487.779905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2487.782689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.789994] Lustre: Skipped 1 previous similar message [ 2487.790869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2487.796600] Lustre: Skipped 1 previous similar message [ 2489.719432] Lustre: server umount lustre-OST0000 complete [ 2498.016469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.040812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2498.044196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.054313] Lustre: Skipped 1 previous similar message [ 2499.683881] Lustre: server umount lustre-MDT0000 complete [ 2503.139656] LustreError: 68770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.165556] LustreError: 68770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2504.325230] LustreError: 68755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777456 with bad export cookie 1860852536398985423 [ 2504.334583] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2504.339814] LustreError: 68755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2504.827682] Lustre: server umount lustre-MDT0001 complete [ 2514.557418] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2516.615907] Key type lgssc unregistered [ 2516.970461] LNet: 73640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2516.987538] LNetError: 73640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2517.013675] LNet: Removed LNI 192.168.202.113@tcp [ 2517.889755] Key type .llcrypt unregistered [ 2517.891862] Key type ._llcrypt unregistered [ 2543.928350] Key type ._llcrypt registered [ 2543.930208] Key type .llcrypt registered [ 2544.010585] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2544.975661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.106802] alg: No test for adler32 (adler32-zlib) [ 2546.189081] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2546.377242] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2548.033725] Key type lgssc registered [ 2549.232382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2557.353273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2557.376632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.043597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.265805] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2569.189603] LustreError: 75052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.208863] LustreError: 75052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2573.977995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.410243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2580.738927] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2587.706617] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2594.613782] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2600.963910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2601.370821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2604.453798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2607.505291] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2615.251724] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2624.304400] Lustre: Failing over lustre-MDT0000 [ 2624.917628] Lustre: server umount lustre-MDT0000 complete [ 2625.506816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2625.515049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.530598] LustreError: 75052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.020095] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.932497] LustreError: 75049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.954870] LustreError: 75049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2631.143671] LustreError: 75052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.169098] LustreError: 75052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2636.257435] LustreError: 75053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.283876] LustreError: 75053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2636.325072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2636.525922] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2636.727269] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2636.856342] LustreError: 78144:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2636.861278] LustreError: 78144:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.113@tcp: cfg command failed: rc = -9 [ 2636.870347] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2636.870347] [ 2636.883270] LustreError: MGC192.168.202.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2636.894683] LustreError: 78132:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2636.900957] LustreError: 78132:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 2636.910399] LustreError: 78132:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2636.914028] LustreError: 78132:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2636.930190] Lustre: server umount lustre-MDT0000 complete [ 2636.935904] LustreError: 78132:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2642.519590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.697621] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2642.983544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2643.064391] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2647.375458] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2647.649081] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2648.041914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2648.076938] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2648.123437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2656.736601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2656.743718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.748957] Lustre: Skipped 1 previous similar message [ 2656.755272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.276244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.285819] Lustre: Skipped 1 previous similar message [ 2662.273760] Lustre: server umount lustre-OST0000 complete [ 2668.515949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.532843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2668.539744] Lustre: Skipped 2 previous similar messages [ 2668.540512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2673.053244] Lustre: server umount lustre-MDT0000 complete [ 2673.642176] LustreError: 75053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.681856] LustreError: 75053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2676.624958] LustreError: 75033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777628 with bad export cookie 4040983268343779552 [ 2676.637869] LustreError: 75033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2676.640434] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2676.950310] Lustre: server umount lustre-MDT0001 complete [ 2687.534079] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2690.572571] Key type lgssc unregistered [ 2690.941941] LNet: 79923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2690.948032] LNetError: 79923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2690.981888] LNet: Removed LNI 192.168.202.113@tcp [ 2691.918512] Key type .llcrypt unregistered [ 2691.920693] Key type ._llcrypt unregistered [ 2705.526909] Key type ._llcrypt registered [ 2705.530294] Key type .llcrypt registered [ 2705.647341] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2705.888705] Key type .llcrypt unregistered [ 2705.891342] Key type ._llcrypt unregistered [ 2730.696834] Key type ._llcrypt registered [ 2730.699979] Key type .llcrypt registered [ 2730.810190] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2731.978286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2732.030653] alg: No test for adler32 (adler32-zlib) [ 2733.031975] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2733.300190] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2734.935245] Key type lgssc registered [ 2736.177747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2744.649857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2744.667399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.383151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2751.892344] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2756.580212] LustreError: 81816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.600820] LustreError: 81816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2760.754267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.307876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2766.656837] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2774.808589] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2780.073667] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2787.009429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2787.461920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.608168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2793.922390] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2800.501073] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2816.482389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2816.491228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2816.506275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2816.515645] Lustre: Skipped 1 previous similar message [ 2816.537479] Lustre: Skipped 1 previous similar message [ 2820.995536] Lustre: server umount lustre-OST0000 complete [ 2826.725621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.727766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2826.740452] Lustre: Skipped 1 previous similar message [ 2831.432086] Lustre: server umount lustre-MDT0000 complete [ 2831.843105] LustreError: 81817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.861557] LustreError: 81817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2835.937082] LustreError: 84306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.968231] LustreError: 84306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2835.994785] LustreError: 81797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777788 with bad export cookie 2359765413153348103 [ 2836.009135] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2836.607388] Lustre: server umount lustre-MDT0001 complete [ 2847.569262] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2850.924820] Key type lgssc unregistered [ 2851.203787] LNet: 85369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2851.210594] LNetError: 85369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2851.221869] LNet: Removed LNI 192.168.202.113@tcp [ 2852.208900] Key type .llcrypt unregistered [ 2852.211556] Key type ._llcrypt unregistered [ 2875.284871] Key type ._llcrypt registered [ 2875.287584] Key type .llcrypt registered [ 2875.360398] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2876.373923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2876.394408] alg: No test for adler32 (adler32-zlib) [ 2877.521344] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2877.813064] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2879.479213] Key type lgssc registered [ 2880.529374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2887.498808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2887.520640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.776081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.233858] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2898.931433] LustreError: 86782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.959213] LustreError: 86782:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2902.028814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.608772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2908.378392] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2916.838839] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2922.292438] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2929.369595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2930.000687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.072183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2937.844188] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2946.171801] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2952.672857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2952.675298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2952.686544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2952.689219] LustreError: Skipped 1 previous similar message [ 2952.714486] Lustre: Skipped 1 previous similar message [ 2954.989352] Lustre: server umount lustre-OST0000 complete [ 2962.929293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2962.954936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2962.964446] Lustre: Skipped 1 previous similar message [ 2962.989404] Lustre: Skipped 2 previous similar messages [ 2966.694990] Lustre: server umount lustre-MDT0000 complete [ 2968.040175] LustreError: 87510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.055729] LustreError: 87510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2970.959631] LustreError: 87518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777923 with bad export cookie 9490451007400094904 [ 2970.971978] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2970.972798] LustreError: 87518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2970.994646] LustreError: 87513:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 2970.998904] LustreError: 87513:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 2971.608926] Lustre: server umount lustre-MDT0001 complete [ 2983.542932] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2986.937199] Key type lgssc unregistered [ 2987.200813] LNet: 90314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2987.205889] LNetError: 90314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2987.215991] LNet: Removed LNI 192.168.202.113@tcp [ 2988.446773] Key type .llcrypt unregistered [ 2988.451281] Key type ._llcrypt unregistered [ 3012.718823] Key type ._llcrypt registered [ 3012.721309] Key type .llcrypt registered [ 3012.847182] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3015.062731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3015.191066] alg: No test for adler32 (adler32-zlib) [ 3016.539752] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 3017.040794] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3018.839232] Key type lgssc registered [ 3021.450699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3031.625777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3031.657619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3033.041845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3037.290233] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3038.190587] LustreError: 91724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.205385] LustreError: 91724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3043.304373] LustreError: 91725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.139409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.645400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3051.493101] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3059.811969] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3067.085925] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3074.542186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3074.788400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.908640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3082.381748] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3090.490765] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3103.719177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3103.732597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.751668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3106.785059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3106.788171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3106.799367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3106.816713] Lustre: Skipped 1 previous similar message [ 3108.433275] Lustre: server umount lustre-OST0000 complete [ 3113.441342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3113.448761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.464592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3117.024662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.026650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3117.051609] Lustre: Skipped 1 previous similar message [ 3118.466310] Lustre: server umount lustre-MDT0000 complete [ 3121.609991] LustreError: 91705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788778073 with bad export cookie 13090495079053422514 [ 3121.620859] LustreError: 91705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3121.623934] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3121.898747] Lustre: server umount lustre-MDT0001 complete [ 3132.455860] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3135.636819] Key type lgssc unregistered [ 3136.009784] LNet: 95257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3136.027915] LNetError: 95257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3136.061883] LNet: Removed LNI 192.168.202.113@tcp [ 3137.142651] Key type .llcrypt unregistered [ 3137.150123] Key type ._llcrypt unregistered [ 3162.060300] Key type ._llcrypt registered [ 3162.064768] Key type .llcrypt registered [ 3162.248271] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3163.546989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3163.611469] alg: No test for adler32 (adler32-zlib) [ 3164.757503] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 3165.016765] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3166.735198] Key type lgssc registered [ 3167.964833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3176.324064] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3176.341193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3192.031748] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3197.152115] LustreError: 96640:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 3223.839258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3230.398996] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3240.162848] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3248.258494] Lustre: server umount lustre-OST0000 complete [ 3261.151332] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 06:50:11 (1788778211) [ 3269.643837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.014674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3274.848097] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3275.236122] LustreError: 98283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.260414] LustreError: 98283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3280.357389] LustreError: 98282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.913816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.242830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3291.367483] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3299.922659] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3307.832250] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3317.491623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3317.832058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3327.850794] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3338.302674] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3340.470904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3356.139270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3356.147651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3356.193470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3360.744952] Lustre: server umount lustre-OST0000 complete [ 3366.375039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.399327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3366.400204] Lustre: Skipped 2 previous similar messages [ 3366.430272] Lustre: Skipped 2 previous similar messages [ 3371.396124] Lustre: server umount lustre-MDT0000 complete [ 3371.495341] LustreError: 98283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.534403] LustreError: 98283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3376.615645] LustreError: 100779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3376.654149] LustreError: 100779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3377.244134] LustreError: 98263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788778329 with bad export cookie 7371669413670769510 [ 3377.261898] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3377.275527] LustreError: 98263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3377.948640] Lustre: server umount lustre-MDT0001 complete [ 3398.420938] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3401.649041] Key type lgssc unregistered [ 3402.113695] LNet: 101997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3402.125768] LNetError: 101997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3402.161234] LNet: Removed LNI 192.168.202.113@tcp [ 3403.207141] Key type .llcrypt unregistered [ 3403.209705] Key type ._llcrypt unregistered [ 3451.906123] Key type ._llcrypt registered [ 3451.907948] Key type .llcrypt registered [ 3452.000414] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3453.459164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3453.512148] alg: No test for adler32 (adler32-zlib) [ 3454.643378] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 3455.097827] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3456.831973] Key type lgssc registered [ 3458.148499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3467.483237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3467.495540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.201443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3474.734777] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3479.525435] LustreError: 104183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3479.543380] LustreError: 104183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3484.656057] LustreError: 104182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3489.534942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3489.776198] LustreError: 104183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.352231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3496.106231] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3504.961829] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3511.443277] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3518.790101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3519.191434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3527.250972] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3537.336721] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3540.661379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3556.325569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3556.336970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3556.337593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3556.339280] Lustre: Skipped 1 previous similar message [ 3559.229920] Lustre: server umount lustre-OST0000 complete [ 3566.563530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.570088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3566.588813] Lustre: Skipped 1 previous similar message [ 3566.608544] Lustre: Skipped 2 previous similar messages [ 3570.127547] Lustre: server umount lustre-MDT0000 complete [ 3571.682646] LustreError: 104183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3571.701813] LustreError: 104183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3575.389226] LustreError: 104162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788778527 with bad export cookie 9384675979533756213 [ 3575.400309] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3575.405853] LustreError: 104162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3575.748709] Lustre: server umount lustre-MDT0001 complete [ 3589.923639] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3593.122953] Key type lgssc unregistered [ 3593.598094] LNet: 107739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3593.606401] LNetError: 107739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3593.639780] LNet: Removed LNI 192.168.202.113@tcp [ 3594.500787] Key type .llcrypt unregistered [ 3594.504145] Key type ._llcrypt unregistered [ 3614.037455] Key type ._llcrypt registered [ 3614.039736] Key type .llcrypt registered [ 3614.169627] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3615.313260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3615.413661] alg: No test for adler32 (adler32-zlib) [ 3616.506974] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 3616.779354] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3618.447246] Key type lgssc registered [ 3619.727748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3629.145795] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3629.162820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3644.767760] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3649.887476] LustreError: 108982:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 3676.703319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3685.882557] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3696.745985] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3704.475958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.754640] LustreError: MGC192.168.202.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3704.767531] LustreError: 110007:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3704.776570] LustreError: 110007:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 3704.798507] LustreError: 110007:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3704.904530] Lustre: server umount lustre-MDT0000 complete [ 3704.909772] LustreError: 110007:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3726.303340] Lustre: 108319:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788778662/real 1788778662] req@00000000b04fa41f x1875670275722624/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788778678 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3726.315168] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3726.322194] LustreError: 108994:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3746.983424] Lustre: server umount lustre-OST0000 complete [ 3757.756513] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 3770.629622] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3817.205236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.473432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3817.504505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3817.595675] Lustre: lustre-MDT0000: new disk, initializing [ 3817.688740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3817.715868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3823.361479] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3835.117922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3835.218922] Lustre: 112804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3835.295537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3835.300887] Lustre: Skipped 1 previous similar message [ 3835.445145] Lustre: lustre-MDT0001: new disk, initializing [ 3835.600465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3835.655347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3835.675987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3841.160546] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3852.635643] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3858.998408] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3865.268885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3865.581191] Lustre: lustre-OST0000: new disk, initializing [ 3865.587570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3865.593537] Lustre: 114773:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3865.695229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3872.570589] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3874.379215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3874.392893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3874.498197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3883.837638] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3887.738683] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3888.043283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3892.144471] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3892.370499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3895.268400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.277319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3895.295513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3895.305303] Lustre: Skipped 1 previous similar message [ 3900.392744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3900.399855] Lustre: Skipped 1 previous similar message [ 3900.858880] Lustre: server umount lustre-OST0000 complete [ 3907.553780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3907.574844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.628309] Lustre: Skipped 1 previous similar message [ 3907.631703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3910.627121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.627750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3910.660345] Lustre: Skipped 1 previous similar message [ 3915.774269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3915.787021] Lustre: Skipped 1 previous similar message [ 3921.376894] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3921.604602] Lustre: server umount lustre-MDT0000 complete [ 3927.641818] LustreError: 113599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788778879 with bad export cookie 10775694388726223572 [ 3927.647961] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3927.656943] LustreError: 113599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3928.015765] Lustre: server umount lustre-MDT0001 complete [ 3938.577981] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 07:01:29 (1788778889) [ 3942.249327] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3960.339704] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 3973.817602] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4016.294162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4016.622330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4016.658941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4016.817345] Lustre: lustre-MDT0000: new disk, initializing [ 4016.959959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4016.990073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4022.578492] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4035.126822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.285502] Lustre: 119690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4035.351690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4035.357409] Lustre: Skipped 1 previous similar message [ 4035.420393] Lustre: lustre-MDT0001: new disk, initializing [ 4035.550257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4035.582788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4035.603951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4041.063797] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4051.876483] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4057.618469] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4064.035136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4064.420142] Lustre: lustre-OST0000: new disk, initializing [ 4064.424226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4064.432762] Lustre: 121660:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4064.536611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4070.041616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4070.051580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4070.161488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4071.225176] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4081.250467] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4086.144505] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4086.351767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4091.202824] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4091.440977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4095.990765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4096.005131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.016672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4096.029249] Lustre: Skipped 3 previous similar messages [ 4099.882152] Lustre: server umount lustre-OST0000 complete [ 4106.211307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4106.232283] Lustre: Skipped 2 previous similar messages [ 4110.425176] Lustre: server umount lustre-MDT0000 complete [ 4111.328162] LustreError: 119703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.352983] LustreError: 119703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4114.426809] LustreError: 119682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779066 with bad export cookie 10775694388726224454 [ 4114.433724] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4114.450177] LustreError: 119682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4114.820813] Lustre: server umount lustre-MDT0001 complete [ 4126.009466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4126.752382] LustreError: 123877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4126.957652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4132.225101] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4132.324743] LustreError: 123878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4136.421662] LustreError: 123877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.561908] LustreError: 123878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.459182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.489846] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4158.116574] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4164.478119] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4171.558196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4171.694945] LustreError: 125701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4179.005726] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4186.865693] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4193.168318] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4202.984767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4202.988589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4203.017268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4203.027528] LustreError: Skipped 2 previous similar messages [ 4203.044344] Lustre: Skipped 3 previous similar messages [ 4204.951213] Lustre: server umount lustre-OST0000 complete [ 4209.632197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4215.661366] Lustre: server umount lustre-MDT0000 complete [ 4218.343776] LustreError: 123877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4218.361578] LustreError: 123877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4219.083504] LustreError: 123858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779171 with bad export cookie 10775694388726225336 [ 4219.085817] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4219.098121] LustreError: 123858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4219.491187] Lustre: server umount lustre-MDT0001 complete [ 4230.064927] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4233.094947] Key type lgssc unregistered [ 4233.512573] LNet: 127507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4233.525101] LNetError: 127507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4233.549692] LNet: Removed LNI 192.168.202.113@tcp [ 4234.774694] Key type .llcrypt unregistered [ 4234.778341] Key type ._llcrypt unregistered [ 4255.245288] Key type ._llcrypt registered [ 4255.249238] Key type .llcrypt registered [ 4255.362461] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 4274.331691] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4276.672970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4276.749650] alg: No test for adler32 (adler32-zlib) [ 4278.214160] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 4278.552117] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 4280.383366] Key type lgssc registered [ 4282.414105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4320.359392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4320.413709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4321.638640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4321.669415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4321.742497] Lustre: lustre-MDT0000: new disk, initializing [ 4321.822973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4321.837534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4326.696712] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4341.590117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.915793] Lustre: 130662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4341.995298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4342.001769] Lustre: Skipped 1 previous similar message [ 4342.172296] Lustre: lustre-MDT0001: new disk, initializing [ 4342.366060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4342.424035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4342.444153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4348.445956] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4361.325304] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4366.533698] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4373.398531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4373.834730] Lustre: lustre-OST0000: new disk, initializing [ 4373.844638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4373.851238] Lustre: 132628:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4373.970765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4380.674614] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4381.792979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4381.803613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4381.921276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4389.979617] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4395.214801] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4395.452423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4400.009286] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4400.247755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4402.669275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4402.670591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4402.680144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4402.686296] Lustre: Skipped 1 previous similar message [ 4402.712318] LustreError: Skipped 1 previous similar message [ 4407.778765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4407.794990] Lustre: Skipped 1 previous similar message [ 4408.381078] Lustre: server umount lustre-OST0000 complete [ 4412.896347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4412.904119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4412.906091] Lustre: Skipped 1 previous similar message [ 4412.931126] Lustre: Skipped 2 previous similar messages [ 4418.019108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4418.032850] Lustre: Skipped 1 previous similar message [ 4418.437787] Lustre: server umount lustre-MDT0000 complete [ 4422.593025] LustreError: 131456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779374 with bad export cookie 16636907147629327391 [ 4422.603524] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4422.606113] LustreError: 131456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4422.993697] Lustre: server umount lustre-MDT0001 complete [ 4432.344506] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 07:09:43 (1788779383) [ 4439.013474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4439.495643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4444.640222] LustreError: 135041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4444.647498] LustreError: 135041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4445.394363] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4449.763197] LustreError: 135042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.904640] LustreError: 135041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4455.172668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4455.749874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4460.591971] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4469.519669] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4475.541883] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4480.231593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4480.261073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4480.270977] Lustre: Skipped 1 previous similar message [ 4483.466582] Lustre: server umount lustre-MDT0000 complete [ 4485.601608] LustreError: 135036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.616099] LustreError: 135036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4487.180835] LustreError: 135023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779439 with bad export cookie 16636907147629328273 [ 4487.196860] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4487.198146] LustreError: 135023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4487.522705] Lustre: server umount lustre-MDT0001 complete [ 4496.528501] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 07:10:47 (1788779447) [ 4503.615483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4519.203848] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4524.319617] LustreError: 137455:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 4550.047465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4556.739470] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4565.222660] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4567.178044] Lustre: server umount lustre-OST0000 complete [ 4575.203443] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 07:12:06 (1788779526) [ 4584.092214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.717251] LustreError: 138901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4584.897264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4590.483391] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4594.149821] LustreError: 138902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4594.162872] LustreError: 138902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4602.425634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.272405] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4615.360372] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4621.606399] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4628.044115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4628.257597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4628.263472] Lustre: Skipped 1 previous similar message [ 4635.966728] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4645.944366] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4659.171947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4659.174080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4659.201171] Lustre: Skipped 3 previous similar messages [ 4659.205871] Lustre: Skipped 1 previous similar message [ 4660.713974] Lustre: server umount lustre-MDT0000 complete [ 4664.295110] LustreError: 138896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.303205] LustreError: 138896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4664.555992] LustreError: 138882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779616 with bad export cookie 16636907147629328938 [ 4664.562301] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4664.565247] LustreError: 138882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4664.884890] Lustre: server umount lustre-MDT0001 complete [ 4678.713818] Lustre: server umount lustre-OST0000 complete [ 4687.715176] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 07:13:59 (1788779639) [ 4694.494614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4695.385495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4700.649310] LustreError: 142577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4700.685544] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4700.688500] LustreError: 142577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4709.769518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4714.448677] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4722.384727] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4727.477476] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4734.905724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4740.360191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4741.747464] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4748.865326] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4754.585053] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4754.837611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4760.360725] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4760.710306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4766.177758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4766.178504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.190825] LustreError: Skipped 1 previous similar message [ 4766.215843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4766.231274] Lustre: Skipped 1 previous similar message [ 4768.711249] Lustre: server umount lustre-OST0000 complete [ 4776.420049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.439064] Lustre: Skipped 2 previous similar messages [ 4787.167476] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4787.464320] Lustre: server umount lustre-MDT0000 complete [ 4791.778092] LustreError: 142577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.802894] LustreError: 142577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4792.174565] LustreError: 142558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779744 with bad export cookie 16636907147629330002 [ 4792.190173] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4792.474180] Lustre: server umount lustre-MDT0001 complete [ 4801.721852] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 07:15:52 (1788779752) [ 4808.863250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4824.415724] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4829.536059] LustreError: 146737:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 4855.328724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4855.337420] Lustre: Skipped 2 previous similar messages [ 4862.898992] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4871.169840] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4879.016600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.124915] LustreError: 128881:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c80abaca x1875670970003072/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4879.665141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4884.251586] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4892.483423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4897.321359] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4904.349068] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4910.932760] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4915.993519] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4916.177986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4921.052430] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4921.372366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4928.484816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4928.488991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4928.513724] Lustre: Skipped 2 previous similar messages [ 4928.518420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4928.520811] Lustre: Skipped 7 previous similar messages [ 4929.842155] Lustre: server umount lustre-MDT0000 complete [ 4933.427734] LustreError: 146743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779885 with bad export cookie 16636907147629331031 [ 4933.443363] LustreError: 146743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4933.457291] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4933.603084] LustreError: 147773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4933.616014] LustreError: 147773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4933.862560] Lustre: server umount lustre-MDT0001 complete [ 4939.986507] Lustre: server umount lustre-OST0000 complete [ 4948.652102] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 07:18:19 (1788779899) [ 4955.548246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4971.233237] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4976.353209] LustreError: 150926:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 5010.339965] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5018.040964] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5026.406119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.271749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5031.907900] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5040.626650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5044.916436] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5052.002358] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5059.001972] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5067.026931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5067.271931] Lustre: lustre-OST0001: new disk, initializing [ 5067.275429] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5067.279984] Lustre: 153801:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5072.970470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5072.988039] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5073.240102] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5075.144098] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5084.113138] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5089.393613] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5089.769406] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5095.038844] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5095.298964] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5097.958197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5097.970636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.988988] Lustre: Skipped 1 previous similar message [ 5097.993955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5098.003939] Lustre: Skipped 1 previous similar message [ 5103.419792] Lustre: server umount lustre-OST0000 complete [ 5109.219266] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5109.234527] LustreError: Skipped 1 previous similar message [ 5113.389737] Lustre: server umount lustre-OST0001 complete [ 5118.435146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5118.440307] LustreError: Skipped 1 previous similar message [ 5123.733658] Lustre: server umount lustre-MDT0000 complete [ 5127.683105] LustreError: 150932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788780079 with bad export cookie 16636907147629331983 [ 5127.686105] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5127.695968] LustreError: 150932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5128.389110] Lustre: server umount lustre-MDT0001 complete [ 5153.747662] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 07:21:45 (1788780105) [ 5155.435083] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5157.085334] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5158.543335] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 07:21:50 (1788780110) [ 5164.391334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.669619] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5164.707411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5164.882305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5164.890072] Lustre: Skipped 6 previous similar messages [ 5168.645772] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5176.475568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5176.571932] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5176.595916] Lustre: 157359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5181.130661] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5188.502108] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5193.648895] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5200.198198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5200.484725] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5206.487979] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5209.291493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5214.684922] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5219.779072] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5220.010070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5224.456495] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5224.666774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5230.063828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5230.068092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5230.085023] Lustre: Skipped 6 previous similar messages [ 5230.121173] Lustre: Skipped 8 previous similar messages [ 5232.963376] Lustre: server umount lustre-OST0000 complete [ 5255.184995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5262.102741] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5269.176966] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5270.915254] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.927820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.932172] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5270.965976] Lustre: Skipped 1 previous similar message [ 5274.538674] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5274.790591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5278.900095] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5279.091339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5283.877124] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95768570d000.ost_server_uuid 50 [ 5285.263252] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95768570d000.ost_server_uuid in FULL state after 0 sec [ 5285.521431] LustreError: 157367:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.13@tcp) returned error from blocking AST (req@00000000884c308e x1875670970136576 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000e16c8fc7/0xe6e22ae7979a8f69 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.13@tcp remote: 0xe9803a83be3c996a expref: 5 pid: 159227 timeout: 5385 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5285.547571] LustreError: lustre-MDT0000: A client on nid 192.168.202.13@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5285.555939] LustreError: 157356:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.13@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000e16c8fc7/0xe6e22ae7979a8f69 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.13@tcp remote: 0xe9803a83be3c996a expref: 6 pid: 159227 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5290.999383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5295.239433] Lustre: server umount lustre-OST0000 complete [ 5306.522779] Lustre: server umount lustre-MDT0000 complete [ 5306.851490] LustreError: 157372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5306.873448] LustreError: 157372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5310.443560] LustreError: 157351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788780262 with bad export cookie 16636907147629333166 [ 5310.451297] LustreError: 157351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5310.453236] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5316.895659] Lustre: server umount lustre-MDT0001 complete [ 5329.347530] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 5333.195417] Key type lgssc unregistered [ 5333.514981] LNet: 163471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5333.526946] LNetError: 163471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5333.541923] LNet: Removed LNI 192.168.202.113@tcp [ 5334.567117] Key type .llcrypt unregistered [ 5334.570590] Key type ._llcrypt unregistered [ 5359.633831] Key type ._llcrypt registered [ 5359.635648] Key type .llcrypt registered [ 5359.709952] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5361.149903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5361.194623] alg: No test for adler32 (adler32-zlib) [ 5362.225259] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 5362.537679] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 5364.295171] Key type lgssc registered [ 5365.737116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5373.771611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5373.813827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5375.576735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5381.313875] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5385.699514] LustreError: 164880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.737559] LustreError: 164880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5390.831248] LustreError: 164881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5390.967037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5391.240254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5397.467691] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5405.295218] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5410.552943] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5418.446504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5419.071356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5420.116334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5426.653267] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5434.526806] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5443.715599] Lustre: Failing over lustre-MDT0000 [ 5444.256679] Lustre: server umount lustre-MDT0000 complete [ 5446.514961] LustreError: 166760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.629903] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.649168] Lustre: Skipped 2 previous similar messages [ 5451.745931] LustreError: 165809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5451.767502] LustreError: 165809:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5456.864685] LustreError: 164880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5456.892855] LustreError: 164880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5457.115529] LustreError: 166724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.13@tcp arrived at 1788780409 with bad export cookie 1906732035927837807 [ 5457.141483] LustreError: 166724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5461.985365] Lustre: 164192:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788780398/real 1788780398] req@0000000056e59a2a x1875672106562560/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788780414 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5462.015557] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.868626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.107761] LustreError: 164875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5467.130031] LustreError: 164875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5472.224236] LustreError: 164190:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000159dae23 x1875672106567552/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5472.511987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5472.576531] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5477.153350] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5477.875048] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5477.881490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5478.969860] Lustre: lustre-MDT0000: Denying connection for new client fca9078a-091b-4a9a-8885-2f5255d91271 (at 192.168.202.13@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5484.367759] Lustre: lustre-MDT0000: Denying connection for new client fca9078a-091b-4a9a-8885-2f5255d91271 (at 192.168.202.13@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 5487.587880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5487.598917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5487.614515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5493.228668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5493.240534] Lustre: Skipped 3 previous similar messages [ 5493.823987] Lustre: server umount lustre-OST0000 complete [ 5498.290308] LustreError: 168615:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5498.304133] Lustre: 167964:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5498.320457] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5498.336580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5498.364615] Lustre: Skipped 1 previous similar message [ 5498.388114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5498.391531] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5498.407242] LustreError: 167964:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008625099e x1875672106582016/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5498.423747] LustreError: 167964:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5498.434757] LustreError: 167964:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5498.450725] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5498.931416] Lustre: server umount lustre-MDT0000 complete [ 5503.457264] LustreError: 164875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5503.483271] LustreError: 164875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5503.520191] LustreError: 164860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788780455 with bad export cookie 1906732035927837912 [ 5503.534762] LustreError: 164860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5503.542659] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5503.560565] LustreError: 164190:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5504.037618] Lustre: server umount lustre-MDT0001 complete [ 5519.708941] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 5522.879824] Key type lgssc unregistered [ 5523.202993] LNet: 169366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5523.227302] LNetError: 169366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5523.258764] LNet: Removed LNI 192.168.202.113@tcp [ 5524.133879] Key type .llcrypt unregistered [ 5524.135522] Key type ._llcrypt unregistered [ 5546.455686] Key type ._llcrypt registered [ 5546.460453] Key type .llcrypt registered [ 5546.572148] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5547.376517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5547.528438] alg: No test for adler32 (adler32-zlib) [ 5548.586828] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 5548.951768] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 5550.712080] Key type lgssc registered [ 5551.823818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5560.642841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5560.686114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5562.453310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5568.466738] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5572.580957] LustreError: 170770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5572.611373] LustreError: 170770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5577.618641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5577.697641] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5577.877711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5582.284890] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5588.979585] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5594.626352] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5601.107256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5601.365235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5607.568116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5607.893358] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5615.699656] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5623.268364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5623.268651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5623.289310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5623.299349] LustreError: Skipped 1 previous similar message [ 5623.304779] Lustre: Skipped 1 previous similar message [ 5624.804231] Lustre: server umount lustre-OST0000 complete [ 5628.898986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5628.907671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.920773] Lustre: Skipped 1 previous similar message [ 5628.924464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5633.510461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5633.512683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5638.628621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5638.638373] Lustre: Skipped 2 previous similar messages [ 5643.233573] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5643.595979] Lustre: server umount lustre-MDT0000 complete [ 5643.746330] LustreError: 170764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5643.763336] LustreError: 170764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5647.096946] LustreError: 170748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788780599 with bad export cookie 2730096559074459468 [ 5647.101807] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5647.115547] LustreError: 170748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5647.401622] Lustre: server umount lustre-MDT0001 complete [ 5657.911767] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 5660.177884] Key type lgssc unregistered [ 5660.437284] LNet: 174300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5660.444147] LNetError: 174300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5660.456879] LNet: Removed LNI 192.168.202.113@tcp [ 5661.360751] Key type .llcrypt unregistered [ 5661.363778] Key type ._llcrypt unregistered [ 5693.242914] Key type ._llcrypt registered [ 5693.246524] Key type .llcrypt registered [ 5693.404676] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5694.513898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5694.622479] alg: No test for adler32 (adler32-zlib) [ 5695.658540] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 5695.857448] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 5697.511191] Key type lgssc registered [ 5698.852595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5707.512749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5707.556683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.060335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5714.448661] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5719.520940] LustreError: 176000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5719.542407] LustreError: 176000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5722.611339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.921741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5728.407481] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5736.206154] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5742.496615] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5749.501103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5749.875380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5753.966223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5757.375639] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5765.893078] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5772.788289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.804579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5774.819471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5774.832051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.849177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5777.892070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5783.018261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5783.035956] Lustre: Skipped 2 previous similar messages [ 5787.103308] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5787.313319] Lustre: server umount lustre-OST0000 complete [ 5793.250992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5793.254788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5793.275371] Lustre: Skipped 1 previous similar message [ 5793.296348] Lustre: Skipped 1 previous similar message [ 5797.536687] Lustre: server umount lustre-MDT0000 complete [ 5798.369939] LustreError: 176728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.408142] LustreError: 176728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5801.364962] LustreError: 175979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788780753 with bad export cookie 13663380838469730980 [ 5801.369589] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5801.380572] LustreError: 175979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5801.599734] Lustre: server umount lustre-MDT0001 complete [ 5812.909690] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 5815.633928] Key type lgssc unregistered [ 5815.937750] LNet: 179531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5815.949278] LNetError: 179531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5815.973800] LNet: Removed LNI 192.168.202.113@tcp [ 5816.931000] Key type .llcrypt unregistered [ 5816.935581] Key type ._llcrypt unregistered [ 5839.649859] Key type ._llcrypt registered [ 5839.653446] Key type .llcrypt registered [ 5839.804115] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5840.914643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5841.152019] alg: No test for adler32 (adler32-zlib) [ 5842.384393] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 5842.647461] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 5844.383221] Key type lgssc registered [ 5845.535258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5856.502773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5856.552667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5858.199803] Lustre: *** cfs_fail_loc=135, val=0*** [ 5858.347663] LustreError: 181054:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5858.356059] LustreError: 181054:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.113@tcp: cfg command failed: rc = -2 [ 5858.362287] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5858.362287] [ 5858.382963] LustreError: MGC192.168.202.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5858.398192] LustreError: 181015:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5858.405388] LustreError: 181015:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5858.411691] LustreError: 181015:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5858.421115] LustreError: 181015:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5858.534552] Lustre: server umount lustre-MDT0000 complete [ 5858.536552] LustreError: 181015:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5865.041703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.545567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.007264] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5875.681591] LustreError: 181427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5875.702832] LustreError: 181427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5878.618906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.060828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5883.698076] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5891.182586] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5897.357216] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5899.747480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5899.759622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5899.772988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5900.769108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.782426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5900.795778] Lustre: Skipped 1 previous similar message [ 5903.905870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5903.911359] Lustre: Skipped 1 previous similar message [ 5905.901841] Lustre: server umount lustre-MDT0000 complete [ 5908.961124] LustreError: 181423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5908.979056] LustreError: 181423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5909.759033] LustreError: 182994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788780861 with bad export cookie 3550524347332606201 [ 5909.767897] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5909.784275] LustreError: 182994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5910.125772] Lustre: server umount lustre-MDT0001 complete [ 5921.451691] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 5924.426434] Key type lgssc unregistered [ 5924.716884] LNet: 183744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5924.725529] LNetError: 183744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5925.801362] LNet: Removed LNI 192.168.202.113@tcp [ 5926.889422] Key type .llcrypt unregistered [ 5926.893129] Key type ._llcrypt unregistered [ 5949.607804] Key type ._llcrypt registered [ 5949.609911] Key type .llcrypt registered [ 5949.725604] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 5949.979449] Key type .llcrypt unregistered [ 5949.984184] Key type ._llcrypt unregistered [ 5966.541986] Key type ._llcrypt registered [ 5966.545197] Key type .llcrypt registered [ 5966.626169] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5967.781220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5967.971456] alg: No test for adler32 (adler32-zlib) [ 5969.043362] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 5969.492601] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 5971.183167] Key type lgssc registered [ 5972.476385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5979.980162] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5979.995472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5995.489730] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6000.607447] LustreError: 185872:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 6027.234227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6033.899343] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6040.507086] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6046.404168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.836901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6046.916537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6051.445968] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6052.333523] LustreError: 185898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.366101] LustreError: 185898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6057.442094] LustreError: 185897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6059.768601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.259693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6065.367653] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6071.854497] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6076.356943] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6078.686192] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6087.652168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6087.662652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.677729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6088.172700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6088.196159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6088.209481] Lustre: Skipped 1 previous similar message [ 6092.711342] Lustre: server umount lustre-OST0000 complete [ 6098.401094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6098.404524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6098.418277] Lustre: Skipped 1 previous similar message [ 6098.438271] Lustre: Skipped 1 previous similar message [ 6102.923068] Lustre: server umount lustre-MDT0000 complete [ 6103.536340] LustreError: 186913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.563439] LustreError: 186913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6106.533146] LustreError: 185878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781058 with bad export cookie 8985575724797238418 [ 6106.545591] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6106.564632] LustreError: 185878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6106.935633] Lustre: server umount lustre-MDT0001 complete [ 6117.879289] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 6120.872978] Key type lgssc unregistered [ 6121.208212] LNet: 189786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6121.223326] LNetError: 189786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6122.279660] LNet: Removed LNI 192.168.202.113@tcp [ 6123.047170] Key type .llcrypt unregistered [ 6123.050387] Key type ._llcrypt unregistered [ 6143.708222] Key type ._llcrypt registered [ 6143.709710] Key type .llcrypt registered [ 6143.786173] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 6144.542421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6144.572794] alg: No test for adler32 (adler32-zlib) [ 6145.671580] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 6145.916686] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 6147.576100] Key type lgssc registered [ 6149.141978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6158.136604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6158.172894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6159.708818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6164.912687] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6169.059277] LustreError: 191175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6169.087400] LustreError: 191175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6172.697207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.256965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6177.245523] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6183.815921] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6189.131974] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6195.143681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6195.638926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6201.713949] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6204.857603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6208.393873] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6212.628357] Lustre: Failing over lustre-MDT0000 [ 6212.898933] Lustre: server umount lustre-MDT0000 complete [ 6214.114885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6214.126345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6214.139740] LustreError: 191174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.139044] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.143300] LustreError: 191903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.162246] Lustre: Skipped 1 previous similar message [ 6215.180986] LustreError: 191903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6220.114427] LustreError: 191169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6225.229234] LustreError: 192465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6225.246832] LustreError: 192465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6231.519255] Lustre: 190485:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788781167/real 1788781167] req@0000000035f8da41 x1875672928642432/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788781183 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6231.538238] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6231.555690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.467451] LustreError: 191171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.486239] LustreError: 191171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6241.767429] LustreError: 190483:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004a7c14f4 x1875672928647808/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6242.124128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6242.199378] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6245.708952] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6246.840948] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6247.414975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6247.439896] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6247.508638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6251.388060] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6259.991994] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6265.312546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6265.321631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.340719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6267.885295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6267.893692] Lustre: Skipped 2 previous similar messages [ 6270.826685] Lustre: server umount lustre-OST0000 complete [ 6278.113397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6278.134670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6278.138870] Lustre: Skipped 1 previous similar message [ 6278.140655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6278.163746] LustreError: Skipped 1 previous similar message [ 6280.786240] Lustre: server umount lustre-MDT0000 complete [ 6283.233307] LustreError: 191171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6283.258889] LustreError: 191171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6284.738070] LustreError: 191157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781236 with bad export cookie 328755377947915848 [ 6284.744344] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6284.745391] LustreError: 191157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6285.249335] Lustre: server umount lustre-MDT0001 complete [ 6296.019776] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 6298.738362] Key type lgssc unregistered [ 6299.151199] LNet: 196109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6299.170179] LNetError: 196109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6299.186573] LNet: Removed LNI 192.168.202.113@tcp [ 6300.032996] Key type .llcrypt unregistered [ 6300.034818] Key type ._llcrypt unregistered [ 6320.921590] Key type ._llcrypt registered [ 6320.923283] Key type .llcrypt registered [ 6321.049889] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 6321.983524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6322.131901] alg: No test for adler32 (adler32-zlib) [ 6323.162237] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 6323.465648] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 6325.144691] Key type lgssc registered [ 6326.422867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6334.057667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6334.069309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.472438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6339.368431] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6345.703651] LustreError: 197496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6345.724895] LustreError: 197496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6347.527399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.782370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6352.062560] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6359.588207] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6365.791963] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6372.348103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6372.878559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6380.092865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6380.382626] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6387.550480] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6390.898620] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6396.777800] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6414.489449] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6423.013552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6423.023934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6423.032368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6423.038682] Lustre: Skipped 1 previous similar message [ 6423.055925] Lustre: Skipped 1 previous similar message [ 6427.610317] Lustre: server umount lustre-OST0000 complete [ 6433.250264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6433.271731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6433.277046] Lustre: Skipped 1 previous similar message [ 6437.885720] Lustre: server umount lustre-MDT0000 complete [ 6438.374894] LustreError: 197496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6438.399512] LustreError: 197496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6442.059810] LustreError: 197475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781394 with bad export cookie 11708681218892402351 [ 6442.064481] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6442.068527] LustreError: 197475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6442.570578] Lustre: server umount lustre-MDT0001 complete [ 6453.515416] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 6455.845270] Key type lgssc unregistered [ 6456.131800] LNet: 201360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6456.151069] LNetError: 201360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6456.167956] LNet: Removed LNI 192.168.202.113@tcp [ 6456.951400] Key type .llcrypt unregistered [ 6456.953270] Key type ._llcrypt unregistered [ 6474.849521] Key type ._llcrypt registered [ 6474.854493] Key type .llcrypt registered [ 6474.985431] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 6476.265842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6476.476587] alg: No test for adler32 (adler32-zlib) [ 6477.579199] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 6477.778321] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 6479.439291] Key type lgssc registered [ 6480.392459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6487.702314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6487.726383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.180992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6494.041855] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6499.300680] LustreError: 202619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6499.324030] LustreError: 202619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6503.378775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6503.740987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6508.127630] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6513.972178] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6518.752842] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6526.012645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6526.816878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6534.086273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6534.892075] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6542.889547] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6548.056860] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6559.349446] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6569.305963] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6574.462434] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6585.312408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6585.316166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.327224] LustreError: Skipped 1 previous similar message [ 6585.329632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6585.343524] Lustre: Skipped 1 previous similar message [ 6585.366388] Lustre: Skipped 1 previous similar message [ 6590.436026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6590.463806] Lustre: Skipped 1 previous similar message [ 6595.560079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6595.568209] Lustre: Skipped 1 previous similar message [ 6597.090693] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6597.319177] Lustre: server umount lustre-OST0000 complete [ 6605.801230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.822982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6605.835278] Lustre: Skipped 1 previous similar message [ 6607.695368] Lustre: server umount lustre-MDT0000 complete [ 6610.903817] LustreError: 202599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781563 with bad export cookie 8407748164867081014 [ 6610.907639] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6610.909866] LustreError: 202599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6611.114498] Lustre: server umount lustre-MDT0001 complete [ 6620.345593] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 6623.599802] Key type lgssc unregistered [ 6624.062307] LNet: 207542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6624.069085] LNetError: 207542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6624.095059] LNet: Removed LNI 192.168.202.113@tcp [ 6625.012861] Key type .llcrypt unregistered [ 6625.014693] Key type ._llcrypt unregistered [ 6651.736423] Key type ._llcrypt registered [ 6651.739355] Key type .llcrypt registered [ 6651.876288] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 6652.916972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6652.984831] alg: No test for adler32 (adler32-zlib) [ 6654.082113] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 6654.322922] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 6655.951170] Key type lgssc registered [ 6657.243801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6665.967931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6666.006973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6667.613987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6671.818640] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6676.965500] LustreError: 208952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6676.975250] LustreError: 208952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6680.481266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6680.798553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6685.335990] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6691.614526] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6696.593593] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6702.609294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6702.976354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6709.387863] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6712.262261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6717.203860] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6732.768751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6732.772332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.775758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6732.793108] Lustre: Skipped 1 previous similar message [ 6737.895076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6737.902272] Lustre: Skipped 2 previous similar messages [ 6738.139173] Lustre: server umount lustre-OST0000 complete [ 6742.498499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6742.502665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6742.511792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6747.925903] Lustre: server umount lustre-MDT0000 complete [ 6748.129657] LustreError: 208952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6748.141488] LustreError: 208952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6751.368836] LustreError: 208932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781703 with bad export cookie 5981644698684326874 [ 6751.379343] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6751.390541] LustreError: 208932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6751.649416] Lustre: server umount lustre-MDT0001 complete [ 6762.922784] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 6765.301672] Key type lgssc unregistered [ 6765.568546] LNet: 212634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6765.576224] LNetError: 212634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6765.594252] LNet: Removed LNI 192.168.202.113@tcp [ 6766.564251] Key type .llcrypt unregistered [ 6766.569259] Key type ._llcrypt unregistered [ 6784.298220] Key type ._llcrypt registered [ 6784.300744] Key type .llcrypt registered [ 6784.410318] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 6785.586274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6785.610681] alg: No test for adler32 (adler32-zlib) [ 6786.736899] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 6787.020230] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 6788.752414] Key type lgssc registered [ 6789.648113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6795.972676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6795.987449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6797.412024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6801.646540] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6807.010306] LustreError: 213875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.033883] LustreError: 213875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6809.398409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6809.882456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6814.135353] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6821.174210] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6826.852529] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6833.102379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6833.624950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6840.511960] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6842.874529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6848.492394] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6865.378763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6865.384374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6865.393334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6868.458194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6868.471190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6868.495327] Lustre: Skipped 1 previous similar message [ 6870.223635] Lustre: server umount lustre-OST0000 complete [ 6876.640274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6876.648121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6876.661785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.690910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.691422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6882.045606] Lustre: server umount lustre-MDT0000 complete [ 6883.809374] LustreError: 213876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6883.821656] LustreError: 213876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6885.885488] LustreError: 215721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781837 with bad export cookie 12920781439502226704 [ 6885.887186] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6885.895476] LustreError: 215721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6886.202761] Lustre: server umount lustre-MDT0001 complete [ 6897.652965] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 6900.728672] Key type lgssc unregistered [ 6901.007896] LNet: 217557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6901.014303] LNetError: 217557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6901.026177] LNet: Removed LNI 192.168.202.113@tcp [ 6902.011142] Key type .llcrypt unregistered [ 6902.014544] Key type ._llcrypt unregistered [ 6920.332911] Key type ._llcrypt registered [ 6920.337771] Key type .llcrypt registered [ 6920.462190] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 6921.712015] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6921.766422] alg: No test for adler32 (adler32-zlib) [ 6922.857502] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 6923.203314] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 6924.871208] Key type lgssc registered [ 6926.123174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6933.544608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6933.565214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6935.003499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6939.040876] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6945.251070] LustreError: 218821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6945.273598] LustreError: 218821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6947.269623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6947.669889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6951.898290] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6959.630386] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6965.777185] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6972.901535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6973.200039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6979.998900] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6980.413596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6987.820357] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6997.477180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6997.519420] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6997.694482] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7003.447438] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7010.262714] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7011.615443] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7023.316041] Lustre: Permanently deactivating lustre-OST0001 [ 7023.336267] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7029.821220] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7039.951732] Lustre: Permanently reactivating lustre-OST0001 [ 7039.992389] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7040.001949] Lustre: Skipped 2 previous similar messages [ 7049.282687] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7049.293994] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7050.639505] Lustre: server umount lustre-OST0001 complete [ 7054.304522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7054.309811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7054.324038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7054.829410] LustreError: 220642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.938507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7059.938901] LustreError: 221375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.942031] Lustre: Skipped 2 previous similar messages [ 7060.576883] Lustre: server umount lustre-OST0000 complete [ 7065.057554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7065.071133] Lustre: Skipped 1 previous similar message [ 7065.076788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7069.663278] LustreError: 222725:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 7070.176275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7070.180200] Lustre: Skipped 1 previous similar message [ 7073.402331] Lustre: server umount lustre-MDT0000 complete [ 7075.296630] LustreError: 219550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7075.317180] LustreError: 219550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7077.700693] LustreError: 218803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782029 with bad export cookie 15633730582547304168 [ 7077.713260] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7077.732100] LustreError: 218803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7078.379777] Lustre: server umount lustre-MDT0001 complete [ 7092.901375] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 7096.667692] Key type lgssc unregistered [ 7096.952553] LNet: 223881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7096.960519] LNetError: 223881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7096.979693] LNet: Removed LNI 192.168.202.113@tcp [ 7098.196457] Key type .llcrypt unregistered [ 7098.200175] Key type ._llcrypt unregistered [ 7141.557736] Key type ._llcrypt registered [ 7141.561176] Key type .llcrypt registered [ 7141.835916] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 7143.453905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7143.484904] alg: No test for adler32 (adler32-zlib) [ 7144.585829] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 7144.867533] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 7146.575295] Key type lgssc registered [ 7148.040508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7157.110814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7157.132272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7158.349981] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7158.365814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7158.540514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7162.563712] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7170.723354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7170.783402] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7170.808902] Lustre: 226082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7171.027381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7175.325253] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7181.941350] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7188.285780] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7196.004292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7196.228993] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7196.409407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7202.753297] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7205.605621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7210.723432] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7214.825158] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7223.078990] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7230.527235] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7236.928391] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7252.412463] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7252.416435] Lustre: Skipped 1 previous similar message [ 7271.538716] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7271.546639] Lustre: Skipped 2 previous similar messages [ 7309.945678] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7309.966450] Lustre: Skipped 4 previous similar messages [ 7379.899830] Lustre: server umount lustre-OST0000 complete [ 7385.057238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.062931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7385.091778] Lustre: Skipped 1 previous similar message [ 7385.103737] Lustre: Skipped 1 previous similar message [ 7390.179468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7390.192691] Lustre: Skipped 1 previous similar message [ 7394.446953] Lustre: server umount lustre-MDT0000 complete [ 7396.194297] Lustre: 225404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788782332/real 1788782332] req@000000007f3b8459 x1875673975272576/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788782348 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7396.252522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7398.571279] LustreError: 226076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782350 with bad export cookie 6287247938598012344 [ 7398.582355] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7398.582858] LustreError: 226076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7398.928410] Lustre: server umount lustre-MDT0001 complete [ 7412.473123] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 7415.104961] Key type lgssc unregistered [ 7415.418565] LNet: 230870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7415.424324] LNetError: 230870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7415.441073] LNet: Removed LNI 192.168.202.113@tcp [ 7416.053148] Key type .llcrypt unregistered [ 7416.054670] Key type ._llcrypt unregistered [ 7444.222377] Key type ._llcrypt registered [ 7444.226372] Key type .llcrypt registered [ 7444.415399] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 7445.400704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7445.417849] alg: No test for adler32 (adler32-zlib) [ 7446.643388] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 7446.979691] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 7448.639176] Key type lgssc registered [ 7450.007335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7458.727647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7458.749189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7460.409690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7465.502802] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7470.562167] LustreError: 232286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7470.586440] LustreError: 232286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7475.068497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7475.404914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7480.218719] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7488.037865] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7494.200954] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7502.014347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7502.759301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7505.839767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7508.681362] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7517.683875] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7539.504173] Lustre: server umount lustre-OST0000 complete [ 7545.824583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.829242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7545.830457] Lustre: Skipped 1 previous similar message [ 7550.946041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7550.952157] Lustre: Skipped 2 previous similar messages [ 7553.175920] Lustre: server umount lustre-MDT0000 complete [ 7556.064102] Lustre: 231595:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788782492/real 1788782492] req@0000000034746313 x1875674292850816/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788782508 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7556.073470] LustreError: 233294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7556.094120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7556.109720] LustreError: 233294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7556.889707] LustreError: 232267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782508 with bad export cookie 13483130511741609793 [ 7556.896273] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7556.899615] LustreError: 232267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7557.474993] Lustre: server umount lustre-MDT0001 complete [ 7571.824798] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 7574.195894] Key type lgssc unregistered [ 7574.464862] LNet: 235985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7574.473827] LNetError: 235985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7574.497554] LNet: Removed LNI 192.168.202.113@tcp [ 7575.429883] Key type .llcrypt unregistered [ 7575.434057] Key type ._llcrypt unregistered [ 7600.318583] Key type ._llcrypt registered [ 7600.321553] Key type .llcrypt registered [ 7600.430168] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 7602.084264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7602.113701] alg: No test for adler32 (adler32-zlib) [ 7603.234687] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 7603.463019] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 7605.211625] Key type lgssc registered [ 7606.818073] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7880.885367] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 7883.747527] Key type lgssc unregistered [ 7884.066579] LNet: 238003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7884.078519] LNetError: 238003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7884.102454] LNet: Removed LNI 192.168.202.113@tcp [ 7885.106645] Key type .llcrypt unregistered [ 7885.109831] Key type ._llcrypt unregistered [ 7912.977739] Key type ._llcrypt registered [ 7912.983590] Key type .llcrypt registered [ 7913.082932] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 7929.473228] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 7930.674572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7930.923276] alg: No test for adler32 (adler32-zlib) [ 7931.992674] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 7932.244414] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 7934.032285] Key type lgssc registered [ 7935.140200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7970.600539] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7612 sec ========== 08:08:40 (1788782920) [ 7973.272577] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:08:43 (1788782923) === [ 7976.804994] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:08:48 (1788782928) === [ 8018.384758] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 8020.989595] Key type lgssc unregistered [ 8021.335450] LNet: 243238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8021.344343] LNetError: 243238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8021.357285] LNet: Removed LNI 192.168.202.113@tcp [ 8022.090641] Key type .llcrypt unregistered [ 8022.092776] Key type ._llcrypt unregistered