[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496052463 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002606] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007659] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010173] LSM: Security Framework initializing [ 0.011069] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013077] *** VALIDATE selinux *** [ 0.022767] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027722] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028193] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032058] *** VALIDATE tmpfs *** [ 0.033587] *** VALIDATE proc *** [ 0.034288] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037023] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038180] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040038] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044510] debug: unmapping init [mem 0xffffffffbda59000-0xffffffffbda60fff] [ 0.047260] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048911] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049033] ... version: 2 [ 0.050018] ... bit width: 48 [ 0.051021] ... generic registers: 4 [ 0.052018] ... value mask: 0000ffffffffffff [ 0.053025] ... max period: 00007fffffffffff [ 0.054021] ... fixed-purpose events: 3 [ 0.055016] ... event mask: 000000070000000f [ 0.056367] rcu: Hierarchical SRCU implementation. [ 0.058987] smp: Bringing up secondary CPUs ... [ 0.059775] x86: Booting SMP configuration: [ 0.060042] .... node #0, CPUs: #1 #2 #3 [ 0.069011] smp: Brought up 1 node, 4 CPUs [ 0.071026] smpboot: Max logical packages: 1 [ 0.072015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.117065] node 0 deferred pages initialised in 43ms [ 0.121229] devtmpfs: initialized [ 0.122382] x86/mm: Memory block size: 128MB [ 0.125203] gcov: version magic: 0x41383552 [ 0.128820] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139643] pinctrl core: initialized pinctrl subsystem [ 0.142215] [ 0.142851] ************************************************************* [ 0.145012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147061] ** ** [ 0.150015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153014] ** ** [ 0.155021] ** This means that this kernel is built to expose internal ** [ 0.158016] ** IOMMU data structures, which may compromise security on ** [ 0.160013] ** your system. ** [ 0.163015] ** ** [ 0.165064] ** If you see this message and you are not debugging the ** [ 0.167000] ** kernel, report this immediately to your vendor! ** [ 0.169014] ** ** [ 0.171015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174014] ************************************************************* [ 0.177431] NET: Registered protocol family 16 [ 0.180763] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183138] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.186153] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.191156] cpuidle: using governor menu [ 0.193814] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196988] PCI: Using configuration type 1 for base access [ 0.200188] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.208077] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.209000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212091] cryptd: max_cpu_qlen set to 1000 [ 0.214386] ACPI: Added _OSI(Module Device) [ 0.217016] ACPI: Added _OSI(Processor Device) [ 0.218015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.220015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226352] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235282] ACPI: Interpreter enabled [ 0.236084] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237184] ACPI: Using IOAPIC for interrupt routing [ 0.238141] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239394] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.250987] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253118] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.256021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260114] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.267111] acpiphp: Slot [2] registered [ 0.269199] acpiphp: Slot [5] registered [ 0.270203] acpiphp: Slot [6] registered [ 0.272134] acpiphp: Slot [7] registered [ 0.274169] acpiphp: Slot [8] registered [ 0.275294] acpiphp: Slot [9] registered [ 0.277296] acpiphp: Slot [10] registered [ 0.280268] acpiphp: Slot [3] registered [ 0.282216] acpiphp: Slot [4] registered [ 0.283162] acpiphp: Slot [11] registered [ 0.285181] acpiphp: Slot [12] registered [ 0.287160] acpiphp: Slot [13] registered [ 0.289175] acpiphp: Slot [14] registered [ 0.290280] acpiphp: Slot [15] registered [ 0.292189] acpiphp: Slot [16] registered [ 0.294138] acpiphp: Slot [17] registered [ 0.295212] acpiphp: Slot [18] registered [ 0.297198] acpiphp: Slot [19] registered [ 0.299186] acpiphp: Slot [20] registered [ 0.300123] acpiphp: Slot [21] registered [ 0.302254] acpiphp: Slot [22] registered [ 0.303117] acpiphp: Slot [23] registered [ 0.305151] acpiphp: Slot [24] registered [ 0.306000] acpiphp: Slot [25] registered [ 0.306000] acpiphp: Slot [26] registered [ 0.306195] acpiphp: Slot [27] registered [ 0.308105] acpiphp: Slot [28] registered [ 0.309187] acpiphp: Slot [29] registered [ 0.311102] acpiphp: Slot [30] registered [ 0.312189] acpiphp: Slot [31] registered [ 0.314118] PCI host bridge to bus 0000:00 [ 0.315037] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.317054] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.319031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.322057] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.325041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.328036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.330303] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.333393] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.336612] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.346835] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.352027] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.354017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358076] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.361626] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.364903] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.367044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.371097] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.392023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.401022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.407892] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.416022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.460016] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.472023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.481019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.517760] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.530023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.538018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.553034] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.562000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.571032] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.579019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.597024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.609919] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.617023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.623028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.647025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.667751] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.678023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.685023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.701020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.709713] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.711378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.714639] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.716449] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.719431] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.725039] iommu: Default domain type: Passthrough [ 0.726244] SCSI subsystem initialized [ 0.727199] ACPI: bus type USB registered [ 0.728103] usbcore: registered new interface driver usbfs [ 0.730066] usbcore: registered new interface driver hub [ 0.731063] usbcore: registered new device driver usb [ 0.732156] pps_core: LinuxPPS API ver. 1 registered [ 0.734011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.737082] PTP clock support registered [ 0.739399] EDAC MC: Ver: 3.0.0 [ 0.741137] PCI: Using ACPI for IRQ routing [ 0.744228] NetLabel: Initializing [ 0.746018] NetLabel: domain hash size = 128 [ 0.747008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.749098] NetLabel: unlabeled traffic allowed by default [ 0.750245] vgaarb: loaded [ 0.751303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.752010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.757000] clocksource: Switched to clocksource kvm-clock [ 0.901569] VFS: Disk quotas dquot_6.6.0 [ 0.903719] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.907642] *** VALIDATE ramfs *** [ 0.909436] *** VALIDATE hugetlbfs *** [ 0.911504] pnp: PnP ACPI init [ 0.914633] pnp: PnP ACPI: found 6 devices [ 0.932179] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.935129] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.937159] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.939025] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.941140] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.943560] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.947242] NET: Registered protocol family 2 [ 0.950194] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.955763] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.959625] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.965544] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.968870] TCP: Hash tables configured (established 65536 bind 65536) [ 0.972877] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.977558] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.980366] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.983352] NET: Registered protocol family 1 [ 0.987368] RPC: Registered named UNIX socket transport module. [ 0.990205] RPC: Registered udp transport module. [ 0.991666] RPC: Registered tcp transport module. [ 0.993328] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.995758] NET: Registered protocol family 44 [ 0.997257] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.999296] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.001532] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.003863] PCI: CLS 0 bytes, default 64 [ 1.005648] Unpacking initramfs... [ 2.783454] debug: unmapping init [mem 0xffff8d8fbcc54000-0xffff8d8fbffbffff] [ 2.789965] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.794167] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.797834] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.477055] Initialise system trusted keyrings [ 3.478859] Key type blacklist registered [ 3.480970] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.491647] zbud: loaded [ 3.495038] *** VALIDATE nfs *** [ 3.496483] *** VALIDATE nfs4 *** [ 3.498157] pstore: using deflate compression [ 3.502644] Platform Keyring initialized [ 3.607773] hrtimer: interrupt took 2780671 ns [ 3.694221] NET: Registered protocol family 38 [ 3.695708] Key type asymmetric registered [ 3.697234] Asymmetric key parser 'x509' registered [ 3.698764] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.701625] io scheduler mq-deadline registered [ 3.703148] io scheduler kyber registered [ 3.705312] io scheduler bfq registered [ 3.707545] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.710965] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.714878] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.718820] ACPI: Power Button [PWRF] [ 3.726321] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.735633] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.753766] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.761768] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.779997] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.823644] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.865562] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.874824] Non-volatile memory driver v1.3 [ 3.877660] Linux agpgart interface v0.103 [ 3.943060] virtio_blk virtio1: [vda] 135992 512-byte logical blocks (69.6 MB/66.4 MiB) [ 3.948624] vda: detected capacity change from 0 to 69627904 [ 3.980065] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.986321] vdb: detected capacity change from 0 to 1073741824 [ 4.019656] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.023796] vdc: detected capacity change from 0 to 2621440000 [ 4.056984] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.064815] vdd: detected capacity change from 0 to 2621440000 [ 4.104289] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.110724] vde: detected capacity change from 0 to 4294967296 [ 4.145866] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.152521] vdf: detected capacity change from 0 to 4294967296 [ 4.166899] libphy: Fixed MDIO Bus: probed [ 4.205503] usbcore: registered new interface driver usbserial_generic [ 4.210295] usbserial: USB Serial support registered for generic [ 4.215070] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.223049] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.227601] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.232444] mousedev: PS/2 mouse device common for all mice [ 4.238872] rtc_cmos 00:05: RTC can wake from S4 [ 4.244512] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.249293] rtc_cmos 00:05: registered as rtc0 [ 4.262650] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.274619] intel_pstate: CPU model not supported [ 4.285667] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.294818] hid: raw HID events driver (C) Jiri Kosina [ 4.310123] usbcore: registered new interface driver usbhid [ 4.316799] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.318475] usbhid: USB HID core driver [ 4.339743] drop_monitor: Initializing network drop monitor service [ 4.347796] Initializing XFRM netlink socket [ 4.351861] NET: Registered protocol family 10 [ 4.360542] Segment Routing with IPv6 [ 4.364781] NET: Registered protocol family 17 [ 4.370686] mpls_gso: MPLS GSO support [ 4.468766] RAS: Correctable Errors collector initialized. [ 4.472860] AVX version of gcm_enc/dec engaged. [ 4.475761] AES CTR mode by8 optimization enabled [ 4.665296] sched_clock: Marking stable (4665197191, 0)->(5760822834, -1095625643) [ 4.677196] registered taskstats version 1 [ 4.679456] Loading compiled-in X.509 certificates [ 4.682460] zswap: loaded using pool lzo/zbud [ 4.724260] Key type big_key registered [ 4.740797] Key type encrypted registered [ 4.743328] ima: No TPM chip found, activating TPM-bypass! [ 4.746512] ima: Allocated hash algorithm: sha1 [ 4.748249] ima: No architecture policies found [ 4.750228] evm: Initialising EVM extended attributes: [ 4.753051] evm: security.selinux [ 4.754771] evm: security.ima [ 4.756403] evm: security.capability [ 4.757989] evm: HMAC attrs: 0x1 [ 4.760946] rtc_cmos 00:05: setting system clock to 2026-06-02 06:07:11 UTC (1780380431) [ 4.770415] debug: unmapping init [mem 0xffffffffbea03000-0xffffffffbebfffff] [ 4.774606] debug: unmapping init [mem 0xffffffffbd782000-0xffffffffbda58fff] [ 4.780181] Write protecting the kernel read-only data: 28672k [ 4.785107] debug: unmapping init [mem 0xffffffffbbe03000-0xffffffffbbffffff] [ 4.788771] debug: unmapping init [mem 0xffffffffbc714000-0xffffffffbc7fffff] [ 4.835801] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.854425] systemd[1]: Detected virtualization kvm. [ 4.857273] systemd[1]: Detected architecture x86-64. [ 4.860433] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.888147] systemd[1]: No hostname configured. [ 4.890152] systemd[1]: Set hostname to . [ 4.892756] random: systemd: uninitialized urandom read (16 bytes read) [ 4.895794] systemd[1]: Initializing machine ID from random generator. [ 4.997231] random: ln: uninitialized urandom read (6 bytes read) [ 5.200812] random: systemd: uninitialized urandom read (16 bytes read) [ 5.204619] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.216941] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.227276] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.382186] device-mapper: uevent: version 1.0.3 [ 6.387202] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.406390] random: fast init done [ 7.471528] virtio_net virtio0 ens2: renamed from eth0 [ 7.822396] scsi host0: ata_piix [ 7.847112] scsi host1: ata_piix [ 7.849861] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.855568] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.824799] random: crng init done [ 13.828316] random: 7 urandom warning(s) missed due to ratelimiting [ 16.161583] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 18.597557] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.567156] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.309238] SELinux: Disabled at runtime. [ 22.378231] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.417162] systemd[1]: Detected virtualization kvm. [ 22.450689] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.591781] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.596611] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.608470] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.614583] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.619506] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.646403] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.678086] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ 23.926008] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 24.795366] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.624851] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.643097] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.135179] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.250062] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 30.622031] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit)[ 31.374585] NFS: Registering the id_resolver key type [ 31.377165] Key type id_resolver registered [ 31.379775] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg245-server login: [ 103.083172] libcfs: loading out-of-tree module taints kernel. [ 103.110291] Key type ._llcrypt registered [ 103.112094] Key type .llcrypt registered [ 103.171185] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 113.676586] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 114.291218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.297937] alg: No test for adler32 (adler32-zlib) [ 115.312174] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 115.636214] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 117.263154] Key type lgssc registered [ 117.868418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 125.761230] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.258780] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 156.521166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.566035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.735506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 157.787520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 157.883213] Lustre: lustre-MDT0000: new disk, initializing [ 157.961398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.976085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.208298] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.728049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.795930] Lustre: 6482:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 170.819108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 170.822204] Lustre: Skipped 1 previous similar message [ 170.880663] Lustre: lustre-MDT0001: new disk, initializing [ 170.912481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.933701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.941922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.131796] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.188974] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 198.009393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.291722] Lustre: lustre-OST0000: new disk, initializing [ 198.294836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 198.299049] Lustre: 8387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.348376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.599568] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.895641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 206.911675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 206.948134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 224.277637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.524698] Lustre: lustre-OST0001: new disk, initializing [ 224.536672] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 224.549563] Lustre: 9444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.655995] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.361943] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.551932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 233.560928] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 233.678357] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 246.313366] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.428671] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 265.774438] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing check_logdir /tmp/testlogs/ [ 271.277243] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing yml_node [ 276.786909] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 279.601838] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 282.378393] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 284.024949] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 2 02:11:49 EDT 2026 [ 303.862617] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 305.449601] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 320.996146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.005784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 321.026224] Lustre: Skipped 2 previous similar messages [ 321.036741] Lustre: Skipped 3 previous similar messages [ 324.067581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.076659] Lustre: Skipped 3 previous similar messages [ 329.184396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.189301] Lustre: Skipped 3 previous similar messages [ 334.316159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.327826] Lustre: Skipped 3 previous similar messages [ 335.327283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 335.600213] Lustre: server umount lustre-MDT0000 complete [ 338.909837] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780380765 with bad export cookie 2074469410839681259 [ 338.913361] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.919944] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 339.423921] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.424840] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 339.437701] Lustre: Skipped 2 previous similar messages [ 339.445370] Lustre: Skipped 1 previous similar message [ 348.578979] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.579832] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.594656] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 348.608618] Lustre: Skipped 3 previous similar messages [ 353.249165] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 353.762876] LustreError: 7742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.775990] LustreError: 7742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 353.912709] Lustre: server umount lustre-MDT0001 complete [ 357.683567] Lustre: server umount lustre-OST0000 complete [ 361.441433] Lustre: server umount lustre-OST0001 complete [ 367.794952] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 376.380511] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 406.316663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.508756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 406.532680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 406.590734] Lustre: lustre-MDT0000: new disk, initializing [ 406.658468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.670694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 411.167700] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 425.241169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.412949] Lustre: 14964:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 425.511795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 425.517301] Lustre: Skipped 1 previous similar message [ 425.674628] Lustre: lustre-MDT0001: new disk, initializing [ 425.845234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 426.031633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 426.075545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 431.221576] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 438.473508] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 442.748331] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 449.881196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 450.195074] Lustre: lustre-OST0000: new disk, initializing [ 450.206982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 450.212145] Lustre: 16705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 450.333909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.231842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 452.257414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 452.393362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 457.686885] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 466.409112] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 472.329296] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 472.587457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 477.114728] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 477.275297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 483.309765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 483.322074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.349466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 483.359528] Lustre: Skipped 1 previous similar message [ 485.559558] Lustre: server umount lustre-OST0000 complete [ 492.512925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 492.523791] LustreError: Skipped 1 previous similar message [ 492.529473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.546556] Lustre: Skipped 1 previous similar message [ 504.287192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 504.489237] Lustre: server umount lustre-MDT0000 complete [ 508.925366] LustreError: 15821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.947618] LustreError: 15821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 509.341068] LustreError: 14955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780380936 with bad export cookie 2074469410839682792 [ 509.355761] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 509.371516] LustreError: 14955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 509.813857] Lustre: server umount lustre-MDT0001 complete [ 513.062553] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 02:15:38 (1780380938) [ 520.174500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.727335] LustreError: 18826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.782817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 524.931261] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 525.794610] LustreError: 18827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.170262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.052579] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 545.882345] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 550.757389] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 558.810114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 566.924135] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 572.502627] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 581.088940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 581.100040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.110182] LustreError: Skipped 1 previous similar message [ 581.112507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 581.112514] Lustre: Skipped 7 previous similar messages [ 581.183215] Lustre: Skipped 2 previous similar messages [ 585.924553] Lustre: server umount lustre-OST0000 complete [ 591.342196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.369613] Lustre: Skipped 1 previous similar message [ 597.834359] Lustre: server umount lustre-MDT0000 complete [ 600.547292] LustreError: 18827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.601071] LustreError: 18827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 602.637887] LustreError: 18806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381029 with bad export cookie 2074469410839683674 [ 602.647477] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 602.664672] LustreError: 18806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 603.151172] Lustre: server umount lustre-MDT0001 complete [ 617.133870] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 619.488890] Key type lgssc unregistered [ 619.706733] LNet: 22004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.733910] LNetError: 22004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 619.749446] LNet: Removed LNI 192.168.202.145@tcp [ 620.683182] Key type .llcrypt unregistered [ 620.687570] Key type ._llcrypt unregistered [ 641.186174] Key type ._llcrypt registered [ 641.188236] Key type .llcrypt registered [ 641.284673] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 642.243590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 642.356030] alg: No test for adler32 (adler32-zlib) [ 643.403775] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 643.661579] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 645.360387] Key type lgssc registered [ 646.576471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 654.987568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 655.011529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.527566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.383889] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 667.112785] LustreError: 23378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 667.126364] LustreError: 23378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 672.123147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.237474] LustreError: 23379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.481337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 679.267770] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 687.039269] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 691.552046] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 699.396945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 699.853265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.972419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 706.181336] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 712.546051] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 730.938725] Lustre: server umount lustre-OST0000 complete [ 737.779094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.785992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 737.797254] Lustre: Skipped 1 previous similar message [ 737.822035] Lustre: Skipped 1 previous similar message [ 742.881643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.887353] Lustre: Skipped 1 previous similar message [ 745.023467] Lustre: server umount lustre-MDT0000 complete [ 748.001743] LustreError: 23378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 748.483930] LustreError: 23359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381175 with bad export cookie 6850573141634888360 [ 748.497878] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 759.108914] Lustre: server umount lustre-MDT0001 complete [ 771.659640] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 774.646176] Key type lgssc unregistered [ 774.930943] LNet: 26999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.936717] LNetError: 26999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.957421] LNet: Removed LNI 192.168.202.145@tcp [ 775.917348] Key type .llcrypt unregistered [ 775.919748] Key type ._llcrypt unregistered [ 801.938262] Key type ._llcrypt registered [ 801.942699] Key type .llcrypt registered [ 802.068846] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 803.833252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 803.977024] alg: No test for adler32 (adler32-zlib) [ 805.407674] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 805.778080] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 807.471162] Key type lgssc registered [ 808.451155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 816.600251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 816.622657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.075184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 822.090374] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 827.361953] LustreError: 28402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 827.379490] LustreError: 28402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 830.174802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.469759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 834.660629] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 840.389974] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 843.997901] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 857.569736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 857.839381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.966074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 865.610523] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 871.331799] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 886.758139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 886.763068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 886.777487] Lustre: Skipped 1 previous similar message [ 886.778179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 891.348973] Lustre: server umount lustre-OST0000 complete [ 896.994636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 896.996823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.024366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.040072] Lustre: Skipped 1 previous similar message [ 897.058786] Lustre: Skipped 2 previous similar messages [ 902.116224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 902.125629] Lustre: Skipped 1 previous similar message [ 902.445471] Lustre: server umount lustre-MDT0000 complete [ 907.145911] LustreError: 28383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381333 with bad export cookie 3660671766725452043 [ 907.148770] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 907.167454] LustreError: 28383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 907.808323] Lustre: server umount lustre-MDT0001 complete [ 920.190592] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 923.366866] Key type lgssc unregistered [ 923.631599] LNet: 31951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.638493] LNetError: 31951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.663952] LNet: Removed LNI 192.168.202.145@tcp [ 924.334337] Key type .llcrypt unregistered [ 924.336101] Key type ._llcrypt unregistered [ 950.510477] Key type ._llcrypt registered [ 950.513390] Key type .llcrypt registered [ 950.650567] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 952.682686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 952.714768] alg: No test for adler32 (adler32-zlib) [ 953.993949] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 954.259462] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 955.907232] Key type lgssc registered [ 957.377155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 966.819555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 966.844836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.303804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 973.220383] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 977.890049] LustreError: 33346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 977.897259] LustreError: 33346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 983.015975] LustreError: 33347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 983.047043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 983.457694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 988.378662] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 994.838535] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 998.821475] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1007.236507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1007.597044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.733112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1015.061154] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1021.999029] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1033.184210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1033.193695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.207298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1033.218020] Lustre: Skipped 1 previous similar message [ 1033.230709] Lustre: Skipped 1 previous similar message [ 1036.674198] Lustre: server umount lustre-OST0000 complete [ 1043.426294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.430172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1043.442468] Lustre: Skipped 1 previous similar message [ 1043.464814] Lustre: Skipped 1 previous similar message [ 1047.250090] Lustre: server umount lustre-MDT0000 complete [ 1048.543809] LustreError: 33347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.552422] LustreError: 33347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1050.707886] LustreError: 35787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381477 with bad export cookie 2381581587194647318 [ 1050.710606] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1050.717507] LustreError: 35787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1050.901822] Lustre: server umount lustre-MDT0001 complete [ 1059.245354] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1061.180167] Key type lgssc unregistered [ 1061.389426] LNet: 36520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.395376] LNetError: 36520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.408302] LNet: Removed LNI 192.168.202.145@tcp [ 1062.003823] Key type .llcrypt unregistered [ 1062.006970] Key type ._llcrypt unregistered [ 1083.051520] Key type ._llcrypt registered [ 1083.055264] Key type .llcrypt registered [ 1083.189225] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1084.224721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1084.291068] alg: No test for adler32 (adler32-zlib) [ 1085.407587] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1085.708555] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1087.418883] Key type lgssc registered [ 1088.845780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1099.289166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1099.309375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.979871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1106.149662] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1110.512779] LustreError: 37913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1110.571638] LustreError: 37913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1115.622862] LustreError: 37914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.491389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.003545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1121.507769] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1127.185712] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1130.815658] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1138.931020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1139.246653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.413752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1146.401265] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1153.808549] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1160.377269] Lustre: lustre-OST0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1160.677283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.684228] Lustre: Skipped 1 previous similar message [ 1160.686685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1165.466679] Lustre: lustre-OST0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1165.485368] Lustre: Skipped 2 previous similar messages [ 1170.613068] Lustre: lustre-OST0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1170.623254] Lustre: Skipped 2 previous similar messages [ 1173.984265] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1174.170647] Lustre: server umount lustre-OST0000 complete [ 1196.516076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.519652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.537502] Lustre: Skipped 1 previous similar message [ 1196.560652] Lustre: Skipped 3 previous similar messages [ 1200.673791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.688832] Lustre: Skipped 1 previous similar message [ 1202.547598] Lustre: server umount lustre-MDT0000 complete [ 1205.730916] LustreError: 37909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.772110] LustreError: 37909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1208.072811] LustreError: 37895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381634 with bad export cookie 2514027260989809638 [ 1208.075834] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1208.090809] LustreError: 37895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1214.554669] Lustre: server umount lustre-MDT0001 complete [ 1226.052366] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1229.876832] Key type lgssc unregistered [ 1230.200576] LNet: 41268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.216762] LNetError: 41268:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.235878] LNet: Removed LNI 192.168.202.145@tcp [ 1231.017910] Key type .llcrypt unregistered [ 1231.021575] Key type ._llcrypt unregistered [ 1255.764850] Key type ._llcrypt registered [ 1255.766548] Key type .llcrypt registered [ 1255.858962] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1256.780078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1256.928373] alg: No test for adler32 (adler32-zlib) [ 1258.595775] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1259.042476] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1260.767371] Key type lgssc registered [ 1262.282528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1272.267379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1272.307741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.770942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.636872] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1284.085683] LustreError: 42672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.094743] LustreError: 42672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1289.186534] LustreError: 42671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.489700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.730581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1294.230258] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1300.517359] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1305.345747] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1315.877869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1316.391354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.483724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1327.168881] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1335.571322] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1345.507062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1345.523509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1345.542685] Lustre: Skipped 2 previous similar messages [ 1345.566696] Lustre: Skipped 2 previous similar messages [ 1348.796364] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1350.592757] Lustre: server umount lustre-MDT0000 complete [ 1350.642238] LustreError: 42668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.685972] LustreError: 42668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1353.878724] LustreError: 43644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.998232] LustreError: 42652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381782 with bad export cookie 5895263042306884063 [ 1356.012102] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1359.001695] LustreError: 42667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.032972] LustreError: 42667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1359.033171] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.45@tcp (stopping) [ 1360.864498] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.479759] Lustre: server umount lustre-MDT0001 complete [ 1378.203923] Lustre: server umount lustre-OST0000 complete [ 1395.238849] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1398.346775] Key type lgssc unregistered [ 1398.746680] LNet: 46179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.763243] LNetError: 46179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1398.805402] LNet: Removed LNI 192.168.202.145@tcp [ 1399.689598] Key type .llcrypt unregistered [ 1399.692183] Key type ._llcrypt unregistered [ 1426.300290] Key type ._llcrypt registered [ 1426.310319] Key type .llcrypt registered [ 1426.492931] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1428.294813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1428.310528] alg: No test for adler32 (adler32-zlib) [ 1429.450943] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1429.826421] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1431.559194] Key type lgssc registered [ 1433.464991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1444.340106] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1444.370739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1460.191452] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1465.378776] LustreError: 47550:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 1492.063271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1499.261488] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1506.293619] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1672.677549] Lustre: server umount lustre-OST0000 complete [ 1694.483890] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1699.431516] Key type lgssc unregistered [ 1699.888823] LNet: 49097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.903438] LNetError: 49097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.925556] LNet: Removed LNI 192.168.202.145@tcp [ 1701.385116] Key type .llcrypt unregistered [ 1701.390614] Key type ._llcrypt unregistered [ 1730.687974] Key type ._llcrypt registered [ 1730.691078] Key type .llcrypt registered [ 1730.841568] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1732.306159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1732.433670] alg: No test for adler32 (adler32-zlib) [ 1733.649381] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1734.029528] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1735.824263] Key type lgssc registered [ 1736.966907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1745.205830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1745.227583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.855182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.025438] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1757.156850] LustreError: 50497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1757.177616] LustreError: 50497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1762.276494] LustreError: 50498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.323417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.953963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1768.331113] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1775.472980] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1779.824739] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1787.362178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1787.896826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.003668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1796.952505] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1805.600639] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1812.450238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1812.451165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1812.469626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1812.497622] Lustre: Skipped 1 previous similar message [ 1817.571838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.589394] Lustre: Skipped 2 previous similar messages [ 1822.692656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1822.704809] Lustre: Skipped 1 previous similar message [ 1823.199152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1823.531942] Lustre: server umount lustre-OST0000 complete [ 1829.350095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1829.358165] LustreError: Skipped 1 previous similar message [ 1829.362335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.380053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1832.928729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.606313] Lustre: server umount lustre-MDT0000 complete [ 1838.050116] LustreError: 50497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1838.070354] LustreError: 50497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1838.581708] LustreError: 50479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382265 with bad export cookie 10958560041293770935 [ 1838.584315] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1838.614907] LustreError: 50479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1839.001839] Lustre: server umount lustre-MDT0001 complete [ 1850.526408] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1854.001562] Key type lgssc unregistered [ 1854.374024] LNet: 53675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.381494] LNetError: 53675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.397566] LNet: Removed LNI 192.168.202.145@tcp [ 1855.399091] Key type .llcrypt unregistered [ 1855.403602] Key type ._llcrypt unregistered [ 1879.117979] Key type ._llcrypt registered [ 1879.120692] Key type .llcrypt registered [ 1879.252599] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1880.704154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1880.765344] alg: No test for adler32 (adler32-zlib) [ 1881.791663] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1882.017836] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1883.695814] Key type lgssc registered [ 1884.724989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1892.612107] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1892.646060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1908.384176] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1913.503223] LustreError: 55044:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 1940.192481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.918900] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1955.233655] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1963.667328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.595788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.694144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1969.632273] LustreError: 55069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.659309] LustreError: 55069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1970.041774] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1974.756112] LustreError: 55070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.772378] LustreError: 55070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1979.875861] LustreError: 55968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.898703] LustreError: 55968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1980.274590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1980.888529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1985.816233] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1991.905958] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1996.700345] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2000.355352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2000.368445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.412696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2001.378755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2001.384606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2001.409043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2001.457418] Lustre: Skipped 1 previous similar message [ 2005.296334] Lustre: server umount lustre-OST0000 complete [ 2032.097801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2032.105261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.152736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2032.169478] Lustre: Skipped 1 previous similar message [ 2036.153124] Lustre: server umount lustre-MDT0000 complete [ 2037.231221] LustreError: 55968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.256418] LustreError: 55968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2040.417614] LustreError: 55050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382467 with bad export cookie 11414059848836660748 [ 2040.425398] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2040.443874] LustreError: 55050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2046.754116] Lustre: server umount lustre-MDT0001 complete [ 2059.285272] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2062.719733] Key type lgssc unregistered [ 2063.018233] LNet: 58436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.038916] LNetError: 58436:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2063.067137] LNet: Removed LNI 192.168.202.145@tcp [ 2063.993793] Key type .llcrypt unregistered [ 2063.996049] Key type ._llcrypt unregistered [ 2095.347125] Key type ._llcrypt registered [ 2095.348777] Key type .llcrypt registered [ 2095.620640] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2097.846266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2097.935529] alg: No test for adler32 (adler32-zlib) [ 2099.340205] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 2100.083830] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2101.959184] Key type lgssc registered [ 2107.560120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2119.118791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2119.132620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.752184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.810889] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2130.919225] LustreError: 59834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.941284] LustreError: 59834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2136.046155] LustreError: 59835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.171762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.055894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2142.462834] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2148.378538] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2153.568603] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2163.187286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.748623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2166.858754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2176.238411] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2185.802843] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2217.953899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2217.965566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.991510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2218.981244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2218.991584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2219.000493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2220.828968] Lustre: server umount lustre-OST0000 complete [ 2225.122227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2225.144706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.172038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2225.174491] Lustre: Skipped 1 previous similar message [ 2229.229788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.237545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2229.249487] Lustre: Skipped 1 previous similar message [ 2231.557944] Lustre: server umount lustre-MDT0000 complete [ 2234.343241] LustreError: 59829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.372996] LustreError: 59829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2235.775623] LustreError: 59816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382662 with bad export cookie 9011963519850961568 [ 2235.784959] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2235.798807] LustreError: 59816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2236.114492] Lustre: server umount lustre-MDT0001 complete [ 2249.219554] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2253.106967] Key type lgssc unregistered [ 2253.389319] LNet: 63040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.408240] LNetError: 63040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.429860] LNet: Removed LNI 192.168.202.145@tcp [ 2254.649149] Key type .llcrypt unregistered [ 2254.651324] Key type ._llcrypt unregistered [ 2283.055084] Key type ._llcrypt registered [ 2283.057708] Key type .llcrypt registered [ 2283.167560] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2284.352696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2284.371932] alg: No test for adler32 (adler32-zlib) [ 2285.413183] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 2285.611326] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2287.311953] Key type lgssc registered [ 2288.332320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2297.208791] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 02:45:22 (1780382722) [ 2304.220528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2304.238896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.725755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.719912] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2316.263970] LustreError: 64728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.305392] LustreError: 64728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2318.300374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.649529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2323.566509] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2330.612339] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2336.304955] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2345.716854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2346.316198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.465640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2353.292593] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2361.773540] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2370.185768] Lustre: Failing over lustre-MDT0000 [ 2370.730849] Lustre: server umount lustre-MDT0000 complete [ 2372.066422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.067745] LustreError: 64723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.096620] Lustre: Skipped 2 previous similar messages [ 2372.097996] LustreError: 64723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2374.865216] LustreError: 64723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.895693] LustreError: 64723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2377.218460] LustreError: 64729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.254415] LustreError: 64729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2381.950365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.043927] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2382.138299] Lustre: *** cfs_fail_loc=135, val=0*** [ 2382.197466] LustreError: 67476:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2382.205189] LustreError: 67476:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.145@tcp: cfg command failed: rc = -2 [ 2382.217618] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2382.217618] [ 2382.231613] LustreError: MGC192.168.202.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2382.252617] LustreError: 67465:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2382.270352] LustreError: 67465:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2382.278672] LustreError: 67465:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2382.289712] LustreError: 67465:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2382.318655] LustreError: 67037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.341366] LustreError: 67037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2382.342584] Lustre: server umount lustre-MDT0000 complete [ 2382.353735] LustreError: 67465:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2390.390645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.600298] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2390.612867] LustreError: 67821:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGC192.168.202.145@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000cc5e00d1) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2390.628983] LustreError: 64720:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2390.919206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.982845] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2395.291304] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2396.001524] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2396.167240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2396.206039] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2396.271093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2404.837460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2404.847604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.864815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2406.372106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2406.373306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.938919] Lustre: server umount lustre-OST0000 complete [ 2416.617670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2416.625865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.630377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2416.630396] Lustre: Skipped 1 previous similar message [ 2416.680361] Lustre: Skipped 1 previous similar message [ 2421.484822] Lustre: server umount lustre-MDT0000 complete [ 2421.730711] LustreError: 64729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.766580] LustreError: 64729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2426.041103] LustreError: 64709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382852 with bad export cookie 3855607209662107807 [ 2426.081951] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2426.841471] Lustre: server umount lustre-MDT0001 complete [ 2441.063741] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2444.410200] Key type lgssc unregistered [ 2444.719973] LNet: 69220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.728457] LNetError: 69220:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2444.770021] LNet: Removed LNI 192.168.202.145@tcp [ 2445.547137] Key type .llcrypt unregistered [ 2445.550901] Key type ._llcrypt unregistered [ 2471.520727] Key type ._llcrypt registered [ 2471.522871] Key type .llcrypt registered [ 2471.621535] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2472.981340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2473.004987] alg: No test for adler32 (adler32-zlib) [ 2474.151785] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 2474.487424] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2476.175766] Key type lgssc registered [ 2477.309753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2485.351869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2485.379549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.938272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.618475] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2496.485473] LustreError: 70618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.500093] LustreError: 70618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2500.823048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.226744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2505.207566] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2511.057514] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2515.351682] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2523.210944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2523.683408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.302798] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2531.892152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2536.225560] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2543.134995] Lustre: Failing over lustre-MDT0000 [ 2543.455892] Lustre: server umount lustre-MDT0000 complete [ 2543.808072] LustreError: 70612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.170640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.177346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2547.196467] LustreError: 70614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.289737] LustreError: 70617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.324915] LustreError: 70617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2555.375477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.596756] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2555.930753] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2556.143707] LustreError: 73364:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2556.160874] LustreError: 73364:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.145@tcp: cfg command failed: rc = -9 [ 2556.171683] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2556.171683] [ 2556.187752] LustreError: MGC192.168.202.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2556.223442] LustreError: 73352:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2556.241444] LustreError: 73352:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 2556.268964] LustreError: 73352:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2556.277686] LustreError: 73352:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2556.322457] Lustre: server umount lustre-MDT0000 complete [ 2556.330393] LustreError: 73352:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2557.423364] LustreError: 70617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.447465] LustreError: 70617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2566.536818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.753699] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2566.765724] LustreError: 70609:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2567.117938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2567.180324] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2569.370209] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2572.249465] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2572.272862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2572.290268] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2572.356872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2581.990791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2581.996541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2582.018759] Lustre: Skipped 2 previous similar messages [ 2582.024175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2587.004585] Lustre: server umount lustre-OST0000 complete [ 2592.740626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2592.742539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.745616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2592.745624] Lustre: Skipped 2 previous similar messages [ 2592.812598] Lustre: Skipped 2 previous similar messages [ 2597.856573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.869824] Lustre: Skipped 2 previous similar messages [ 2598.827195] Lustre: server umount lustre-MDT0000 complete [ 2602.981233] LustreError: 70617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.010112] LustreError: 70617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2603.668366] LustreError: 73743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383030 with bad export cookie 14533275094479524421 [ 2603.673705] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.682892] LustreError: 73743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2604.015542] Lustre: server umount lustre-MDT0001 complete [ 2618.654612] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2621.924561] Key type lgssc unregistered [ 2622.217682] LNet: 75113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2622.231140] LNetError: 75113:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.255316] LNet: Removed LNI 192.168.202.145@tcp [ 2623.363196] Key type .llcrypt unregistered [ 2623.366872] Key type ._llcrypt unregistered [ 2635.365347] Key type ._llcrypt registered [ 2635.373662] Key type .llcrypt registered [ 2635.515438] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2635.742747] Key type .llcrypt unregistered [ 2635.749669] Key type ._llcrypt unregistered [ 2661.232571] Key type ._llcrypt registered [ 2661.234338] Key type .llcrypt registered [ 2661.354882] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2662.331623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2662.352919] alg: No test for adler32 (adler32-zlib) [ 2663.453821] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 2663.725697] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2665.401888] Key type lgssc registered [ 2666.802242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2675.627363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2675.636900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.193150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.226310] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2686.442200] LustreError: 76981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.483167] LustreError: 76981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2691.559834] LustreError: 76980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.484506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.067480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2700.062181] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2706.947484] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2710.904869] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2717.265655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2717.654221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.795555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2725.141340] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2730.823432] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2740.196714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2740.202569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.218403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2742.244141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2742.246670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2742.247965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2745.704306] Lustre: server umount lustre-OST0000 complete [ 2751.456838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2751.460288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.492387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2751.499659] Lustre: Skipped 1 previous similar message [ 2755.707175] Lustre: server umount lustre-MDT0000 complete [ 2757.608603] LustreError: 77940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2757.641620] LustreError: 77940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2759.247767] LustreError: 76961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383185 with bad export cookie 6328319990835951846 [ 2759.257224] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2759.264984] LustreError: 76961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2759.706679] Lustre: server umount lustre-MDT0001 complete [ 2773.455104] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2777.577705] Key type lgssc unregistered [ 2778.165876] LNet: 80160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2778.194317] LNetError: 80160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2779.239734] LNet: Removed LNI 192.168.202.145@tcp [ 2780.338860] Key type .llcrypt unregistered [ 2780.341905] Key type ._llcrypt unregistered [ 2809.346077] Key type ._llcrypt registered [ 2809.347715] Key type .llcrypt registered [ 2809.470766] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2810.581127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2810.881633] alg: No test for adler32 (adler32-zlib) [ 2812.059959] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 2812.318915] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2814.039211] Key type lgssc registered [ 2815.443537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2823.390825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2823.430837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.079037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2829.401155] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2834.412902] LustreError: 81557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2834.434662] LustreError: 81557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2837.962328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.445336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2842.849203] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2848.886421] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2852.647723] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2860.472478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2861.070393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.204856] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2869.312698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2873.985465] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2882.146293] Lustre: server umount lustre-OST0000 complete [ 2889.699108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.703076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2889.716019] Lustre: Skipped 1 previous similar message [ 2889.717113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2889.728595] Lustre: Skipped 1 previous similar message [ 2894.816753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.834041] Lustre: Skipped 1 previous similar message [ 2896.989098] Lustre: server umount lustre-MDT0000 complete [ 2899.942931] LustreError: 81558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.980273] LustreError: 81558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2900.771738] Lustre: 80869:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780383311/real 1780383311] req@0000000057a1c7bf x1866867135243392/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780383327 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2900.797444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.149550] LustreError: 81537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383327 with bad export cookie 14215852824873926596 [ 2901.160557] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2901.624981] Lustre: server umount lustre-MDT0001 complete [ 2915.149654] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2917.839968] Key type lgssc unregistered [ 2918.094262] LNet: 84733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2918.110204] LNetError: 84733:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2919.142529] LNet: Removed LNI 192.168.202.145@tcp [ 2920.179339] Key type .llcrypt unregistered [ 2920.181870] Key type ._llcrypt unregistered [ 2944.826424] Key type ._llcrypt registered [ 2944.828978] Key type .llcrypt registered [ 2944.932827] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2946.170389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2946.192236] alg: No test for adler32 (adler32-zlib) [ 2947.259927] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 2947.454039] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2949.135201] Key type lgssc registered [ 2950.212966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2959.012541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2959.039565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2960.555058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.343186] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2971.106211] LustreError: 86129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.124425] LustreError: 86129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2973.820669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.164156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2978.578994] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2984.119628] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2988.466393] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2995.553230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2995.919910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3001.899181] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3006.198757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3008.443470] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3019.750175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3019.760053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.810510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3020.781134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3020.793089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.815863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.289938] Lustre: server umount lustre-OST0000 complete [ 3026.914048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.939613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3026.956550] Lustre: Skipped 1 previous similar message [ 3032.032568] LustreError: 86125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.078753] LustreError: 86125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3032.216707] Lustre: server umount lustre-MDT0000 complete [ 3036.365257] LustreError: 86109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383463 with bad export cookie 2641904365673494842 [ 3036.372022] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3036.379744] LustreError: 86109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3036.973461] Lustre: server umount lustre-MDT0001 complete [ 3049.492405] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3052.513420] Key type lgssc unregistered [ 3052.847270] LNet: 89306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3052.867137] LNetError: 89306:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3052.893621] LNet: Removed LNI 192.168.202.145@tcp [ 3053.974116] Key type .llcrypt unregistered [ 3053.976661] Key type ._llcrypt unregistered [ 3081.354828] Key type ._llcrypt registered [ 3081.359057] Key type .llcrypt registered [ 3081.522959] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3082.941620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3083.024368] alg: No test for adler32 (adler32-zlib) [ 3084.060975] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 3084.233842] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3085.999170] Key type lgssc registered [ 3087.006903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3094.614299] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3094.637152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.239363] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3115.361871] LustreError: 90677:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 3142.111553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3148.811690] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3155.543638] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3163.352841] Lustre: server umount lustre-OST0000 complete [ 3173.318673] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:59:58 (1780383598) [ 3181.144689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.702992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.957378] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3187.180944] LustreError: 92205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3187.195368] LustreError: 92205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3191.266252] LustreError: 92204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.922644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.422451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3200.774359] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3207.114901] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3211.873800] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3218.860421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3219.363081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.879558] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3226.543929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3232.209500] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3246.048752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3246.062251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.096585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3247.081203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3247.100412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3247.100447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.129283] Lustre: Skipped 1 previous similar message [ 3249.565743] Lustre: server umount lustre-OST0000 complete [ 3256.800621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3256.810623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3256.838522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.750295] Lustre: server umount lustre-MDT0000 complete [ 3262.437694] LustreError: 92205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.452930] LustreError: 92205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3263.979145] LustreError: 92186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383690 with bad export cookie 12540443751662158708 [ 3263.981640] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3264.002773] LustreError: 92186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3264.401701] Lustre: server umount lustre-MDT0001 complete [ 3276.820613] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3279.347318] Key type lgssc unregistered [ 3279.668530] LNet: 95553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3279.686457] LNetError: 95553:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3279.714878] LNet: Removed LNI 192.168.202.145@tcp [ 3280.512390] Key type .llcrypt unregistered [ 3280.514918] Key type ._llcrypt unregistered [ 3323.429964] Key type ._llcrypt registered [ 3323.431918] Key type .llcrypt registered [ 3323.504782] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3324.853130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3324.947331] alg: No test for adler32 (adler32-zlib) [ 3326.125961] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 3326.430342] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3328.135346] Key type lgssc registered [ 3329.357896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3336.240825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3336.261822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.832686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.870728] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3347.426856] LustreError: 97720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.444644] LustreError: 97720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3350.285523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.625674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3354.130861] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3359.205273] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3362.860221] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3370.099773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3370.608460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.499310] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3382.945858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3385.391578] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3396.590506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3396.613267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.631870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3398.634779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3398.652429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.654084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3402.487127] Lustre: server umount lustre-OST0000 complete [ 3406.826830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3406.845720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.859146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3406.867258] Lustre: Skipped 1 previous similar message [ 3408.863994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3408.871607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3412.802604] Lustre: server umount lustre-MDT0000 complete [ 3413.984578] LustreError: 98906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.003631] LustreError: 98906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3416.715459] LustreError: 99875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383843 with bad export cookie 10361760555205042426 [ 3416.725131] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3416.727850] LustreError: 99875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3417.125536] Lustre: server umount lustre-MDT0001 complete [ 3427.485869] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3430.579040] Key type lgssc unregistered [ 3430.904603] LNet: 100899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3430.914473] LNetError: 100899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3430.930407] LNet: Removed LNI 192.168.202.145@tcp [ 3431.698749] Key type .llcrypt unregistered [ 3431.711126] Key type ._llcrypt unregistered [ 3451.497694] Key type ._llcrypt registered [ 3451.501144] Key type .llcrypt registered [ 3451.603060] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3452.691918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3452.759886] alg: No test for adler32 (adler32-zlib) [ 3453.916630] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 3454.163673] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3455.911246] Key type lgssc registered [ 3457.182036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3465.551417] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3465.604527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3481.311650] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3486.431400] LustreError: 102125:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 3513.119190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3520.620408] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3528.250537] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3537.169517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.511766] LustreError: MGC192.168.202.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3537.527858] LustreError: 103037:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3537.543193] LustreError: 103037:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3537.557448] LustreError: 103037:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3537.642472] Lustre: server umount lustre-MDT0000 complete [ 3537.646459] LustreError: 103037:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3555.432296] Lustre: server umount lustre-OST0000 complete [ 3567.246357] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 3580.961798] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3619.787914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.121292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3620.164262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3620.269483] Lustre: lustre-MDT0000: new disk, initializing [ 3620.404127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3620.430616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3626.519189] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3641.762445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.906612] Lustre: 105793:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3641.956238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3641.960166] Lustre: Skipped 1 previous similar message [ 3642.018652] Lustre: lustre-MDT0001: new disk, initializing [ 3642.099662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3642.162144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3642.174264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3647.245814] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3655.445583] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3659.520879] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3666.097161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3666.330682] Lustre: lustre-OST0000: new disk, initializing [ 3666.334710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3666.339555] Lustre: 107538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3666.445302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.857960] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3675.820431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3675.826412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3675.965389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3680.299930] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3685.197319] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3685.618662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3689.112499] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3689.302209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3696.608829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3696.625115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.634975] LustreError: Skipped 1 previous similar message [ 3696.641342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.656799] Lustre: Skipped 1 previous similar message [ 3696.670784] Lustre: Skipped 1 previous similar message [ 3697.919879] Lustre: server umount lustre-OST0000 complete [ 3703.785060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3703.795819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3703.823581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3706.849272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3706.852445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3709.850148] Lustre: server umount lustre-MDT0000 complete [ 3713.545988] LustreError: 105788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384140 with bad export cookie 9974865467992701511 [ 3713.555770] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3713.562186] LustreError: 105788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3714.046765] Lustre: server umount lustre-MDT0001 complete [ 3722.901162] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 03:09:08 (1780384148) [ 3726.293231] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3739.979850] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 3748.272770] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3783.681466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.895845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3783.926316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3784.010240] Lustre: lustre-MDT0000: new disk, initializing [ 3784.114851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3784.143374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3788.602512] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3798.531189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3798.651522] Lustre: 112257:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3798.699582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3798.706101] Lustre: Skipped 1 previous similar message [ 3798.766514] Lustre: lustre-MDT0001: new disk, initializing [ 3798.850824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3798.878584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3798.898400] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3803.562829] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3812.060994] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3815.738519] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3823.082809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3823.444741] Lustre: lustre-OST0000: new disk, initializing [ 3823.457618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3823.463967] Lustre: 113998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3823.650685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.441500] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3832.397641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3832.420267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3832.622935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3837.793979] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3842.676666] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3842.995879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3847.244388] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3847.409251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3853.280954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3853.294790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3853.308265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.321701] Lustre: Skipped 1 previous similar message [ 3853.340207] Lustre: Skipped 2 previous similar messages [ 3855.653784] Lustre: server umount lustre-OST0000 complete [ 3860.459565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3860.464434] LustreError: Skipped 1 previous similar message [ 3860.469822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.492665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3865.794546] Lustre: server umount lustre-MDT0000 complete [ 3868.644898] LustreError: 112266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3868.656433] LustreError: 112266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3869.462136] LustreError: 112251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384296 with bad export cookie 9974865467992702393 [ 3869.471610] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3869.480460] LustreError: 112251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3869.802736] Lustre: server umount lustre-MDT0001 complete [ 3878.875890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.499162] LustreError: 116068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3879.624212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3884.248080] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3885.043126] LustreError: 116069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.151260] LustreError: 116068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3892.467359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3897.451446] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3903.255439] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3907.922298] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3914.176623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3919.561839] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3924.993439] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3935.597469] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3941.345346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3941.354165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.359691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3941.380480] Lustre: Skipped 2 previous similar messages [ 3941.384946] Lustre: Skipped 2 previous similar messages [ 3945.239939] Lustre: server umount lustre-OST0000 complete [ 3961.830550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3961.838041] Lustre: Skipped 6 previous similar messages [ 3964.383236] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3964.911469] Lustre: server umount lustre-MDT0000 complete [ 3966.951156] LustreError: 117048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3966.988659] LustreError: 117048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3968.663517] LustreError: 116048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384395 with bad export cookie 9974865467992703275 [ 3968.666134] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3968.677967] LustreError: 116048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3969.171727] Lustre: server umount lustre-MDT0001 complete [ 3979.771616] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3983.044573] Key type lgssc unregistered [ 3983.332737] LNet: 119342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3983.344076] LNetError: 119342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3983.374331] LNet: Removed LNI 192.168.202.145@tcp [ 3984.240314] Key type .llcrypt unregistered [ 3984.246603] Key type ._llcrypt unregistered [ 4002.946556] Key type ._llcrypt registered [ 4002.948228] Key type .llcrypt registered [ 4003.065487] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 4016.952614] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4018.142530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4018.236317] alg: No test for adler32 (adler32-zlib) [ 4019.344675] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 4019.547267] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4021.287164] Key type lgssc registered [ 4022.380957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4054.768747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4054.782834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4055.998910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4056.037644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4056.165852] Lustre: lustre-MDT0000: new disk, initializing [ 4056.316962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4056.333401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4060.078218] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4069.866523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.046251] Lustre: 122465:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4070.132749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4070.137824] Lustre: Skipped 1 previous similar message [ 4070.344929] Lustre: lustre-MDT0001: new disk, initializing [ 4070.490491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4070.533053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4070.555920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4075.027400] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4081.664280] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4085.432911] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4092.086745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4092.351767] Lustre: lustre-OST0000: new disk, initializing [ 4092.356859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4092.363161] Lustre: 124204:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4092.491300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4098.192933] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4100.186686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4100.214792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4100.312280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4105.838117] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4110.564075] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4110.736366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4114.828761] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4115.055287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4121.062431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4121.065961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.071752] LustreError: Skipped 1 previous similar message [ 4121.083142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4121.097920] Lustre: Skipped 1 previous similar message [ 4126.178881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4126.183603] Lustre: Skipped 1 previous similar message [ 4131.295180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4131.303407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4131.317971] Lustre: Skipped 1 previous similar message [ 4131.629706] Lustre: server umount lustre-OST0000 complete [ 4136.427777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.431591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4136.445676] Lustre: Skipped 2 previous similar messages [ 4136.462060] Lustre: Skipped 1 previous similar message [ 4141.544730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4141.562631] Lustre: Skipped 1 previous similar message [ 4141.995465] Lustre: server umount lustre-MDT0000 complete [ 4145.934532] LustreError: 123242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384572 with bad export cookie 6534194434364506324 [ 4145.941377] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4145.953459] LustreError: 123242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4146.267312] Lustre: server umount lustre-MDT0001 complete [ 4155.693219] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 03:16:21 (1780384581) [ 4162.318130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4162.743171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4166.869814] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4168.170394] LustreError: 126470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4168.180182] LustreError: 126470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4173.288725] LustreError: 126471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4175.822173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.376294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4181.216551] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4186.653288] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4189.824177] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4193.760885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4193.800205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4193.814793] Lustre: Skipped 1 previous similar message [ 4198.500889] Lustre: server umount lustre-MDT0000 complete [ 4198.881681] LustreError: 126467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4198.910971] LustreError: 126467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4201.966340] LustreError: 126465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4201.975104] LustreError: 126465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4202.307500] LustreError: 126452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384629 with bad export cookie 6534194434364507206 [ 4202.313378] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4202.323910] LustreError: 126452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4202.602582] Lustre: server umount lustre-MDT0001 complete [ 4210.457374] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 03:17:16 (1780384636) [ 4215.967965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4231.587650] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4236.703854] LustreError: 128657:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 4262.431616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.612377] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4274.879938] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4277.001349] Lustre: server umount lustre-OST0000 complete [ 4284.853053] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 03:18:30 (1780384710) [ 4291.371457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4291.864241] LustreError: 129989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4291.946662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4295.533450] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4301.287212] LustreError: 129990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.310355] LustreError: 129990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4304.145979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4308.572422] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4313.758787] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4316.910859] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4322.594840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.605957] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4332.824828] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4345.827082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4345.838186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.852174] Lustre: Skipped 1 previous similar message [ 4345.858409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4350.502217] Lustre: server umount lustre-MDT0000 complete [ 4352.996340] LustreError: 129989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.027768] LustreError: 129989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4354.118685] LustreError: 131608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384780 with bad export cookie 6534194434364507871 [ 4354.120421] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4354.131902] LustreError: 131608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4354.543401] Lustre: server umount lustre-MDT0001 complete [ 4368.447923] Lustre: server umount lustre-OST0000 complete [ 4377.392771] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 03:20:02 (1780384802) [ 4384.722674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4385.416538] LustreError: 133306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4385.519971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4385.526738] Lustre: Skipped 2 previous similar messages [ 4389.513727] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4397.290890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4401.358259] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4406.387123] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4410.360512] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4416.165744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4422.197406] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4426.947217] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4430.703976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4431.107789] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4431.356460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4434.847055] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4434.983750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4437.984254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4437.986702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.998318] Lustre: Skipped 2 previous similar messages [ 4438.005769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4438.012632] Lustre: Skipped 3 previous similar messages [ 4442.791082] Lustre: server umount lustre-OST0000 complete [ 4448.736990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4448.744119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.750804] Lustre: Skipped 1 previous similar message [ 4452.629955] Lustre: server umount lustre-MDT0000 complete [ 4456.276671] LustreError: 134025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384883 with bad export cookie 6534194434364508942 [ 4456.283250] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4456.293546] LustreError: 134025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4456.557935] Lustre: server umount lustre-MDT0001 complete [ 4463.661357] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 03:21:29 (1780384889) [ 4469.150495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4484.831610] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4489.952768] LustreError: 137089:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 4515.679813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4515.683738] Lustre: Skipped 2 previous similar messages [ 4521.000720] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4524.884141] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4530.134020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4530.469760] LustreError: 137114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4530.486386] LustreError: 137114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4530.609459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4533.994894] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4541.127856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4544.535609] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4548.543489] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4551.394143] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4555.120527] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4555.293870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4558.608837] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4558.781830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4561.889698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4561.897727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4561.897906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4561.915454] Lustre: Skipped 2 previous similar messages [ 4561.943450] Lustre: Skipped 7 previous similar messages [ 4566.778841] Lustre: server umount lustre-MDT0000 complete [ 4569.595764] LustreError: 137096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384996 with bad export cookie 6534194434364509964 [ 4569.604928] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4569.607215] LustreError: 137096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4569.924577] Lustre: server umount lustre-MDT0001 complete [ 4582.580267] Lustre: server umount lustre-OST0000 complete [ 4588.865389] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 03:23:34 (1780385014) [ 4594.189913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4609.695654] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4614.815952] LustreError: 140905:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 4644.379533] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4649.004763] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4655.067042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4655.407517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4655.413945] Lustre: Skipped 3 previous similar messages [ 4655.470985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4658.856380] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4660.705895] LustreError: 140931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4660.723142] LustreError: 140931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4665.930488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.376205] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4674.655864] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4677.673860] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4683.127504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4683.283534] Lustre: lustre-OST0001: new disk, initializing [ 4683.287275] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4683.292862] Lustre: 143436:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4688.510378] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4689.930987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4689.942433] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4690.018834] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4694.558207] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4698.345133] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4698.514578] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4702.035365] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4702.176405] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4705.250240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4705.260451] Lustre: Skipped 1 previous similar message [ 4705.267581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4705.286615] Lustre: Skipped 1 previous similar message [ 4718.048132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4718.125156] Lustre: server umount lustre-OST0000 complete [ 4725.735262] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4725.746721] LustreError: Skipped 1 previous similar message [ 4727.714879] Lustre: server umount lustre-OST0001 complete [ 4736.844273] Lustre: server umount lustre-MDT0000 complete [ 4740.071248] LustreError: 140912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385166 with bad export cookie 6534194434364510916 [ 4740.078767] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4740.080628] LustreError: 140912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4740.349405] Lustre: server umount lustre-MDT0001 complete [ 4758.535620] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 03:26:24 (1780385184) [ 4760.392902] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4762.304744] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4763.817516] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 03:26:29 (1780385189) [ 4770.186498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.474582] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4770.512317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4773.715752] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4779.926528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4779.982576] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4779.998114] Lustre: 146850:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4783.645407] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4788.075431] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4790.787877] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4796.342176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4796.558600] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4802.766398] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4806.768862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4807.359984] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4810.885489] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4811.197195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4814.870282] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4815.092100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4817.382439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4817.395042] Lustre: Skipped 4 previous similar messages [ 4817.395826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4831.200506] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4831.323794] Lustre: server umount lustre-OST0000 complete [ 4849.707862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4854.463656] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4858.162721] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4858.178142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4858.183964] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4858.198690] Lustre: Skipped 1 previous similar message [ 4858.943844] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4862.580644] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4862.703449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4865.654762] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4865.814675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4868.216600] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff97e1467b8800.ost_server_uuid 50 [ 4869.218780] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff97e1467b8800.ost_server_uuid in FULL state after 0 sec [ 4869.394507] LustreError: 146859:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.45@tcp) returned error from blocking AST (req@000000005b813dc6 x1866868401089280 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f63394f4/0x5aae1f37e9a19c1e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.45@tcp remote: 0x9db49b4666450ca1 expref: 5 pid: 148987 timeout: 4969 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4869.427790] LustreError: lustre-MDT0000: A client on nid 192.168.202.45@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4869.437829] LustreError: 146848:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.45@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f63394f4/0x5aae1f37e9a19c1e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.45@tcp remote: 0x9db49b4666450ca1 expref: 6 pid: 148987 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4873.700168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4878.694712] Lustre: server umount lustre-OST0000 complete [ 4887.633392] Lustre: server umount lustre-MDT0000 complete [ 4890.903208] LustreError: 146844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385317 with bad export cookie 6534194434364512099 [ 4890.907320] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4890.913625] LustreError: 146844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4897.183703] Lustre: server umount lustre-MDT0001 complete [ 4904.143838] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4906.117058] Key type lgssc unregistered [ 4906.384178] LNet: 152326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4906.393537] LNetError: 152326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4906.414391] LNet: Removed LNI 192.168.202.145@tcp [ 4906.964395] Key type .llcrypt unregistered [ 4906.966902] Key type ._llcrypt unregistered [ 4929.849239] Key type ._llcrypt registered [ 4929.852394] Key type .llcrypt registered [ 4929.969420] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4932.479547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4932.547139] alg: No test for adler32 (adler32-zlib) [ 4933.550794] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 4933.768968] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4935.463260] Key type lgssc registered [ 4937.737443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4949.253522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4949.267392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.567090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.402153] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4960.760298] LustreError: 153730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4960.811784] LustreError: 153730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4965.864409] LustreError: 153729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4970.988450] LustreError: 153730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4972.167240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4972.517847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4976.867584] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4982.338369] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4986.706673] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4993.914384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4994.248164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4995.332048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5000.391827] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5005.828514] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5012.116627] Lustre: Failing over lustre-MDT0000 [ 5012.467471] Lustre: server umount lustre-MDT0000 complete [ 5012.974262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.984276] LustreError: 153726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5012.996069] Lustre: Skipped 2 previous similar messages [ 5013.037374] LustreError: 153726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5018.088543] LustreError: 153730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5018.103243] LustreError: 153730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5024.934163] LustreError: 153711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.45@tcp arrived at 1780385451 with bad export cookie 6774311799569871695 [ 5024.959712] LustreError: 153711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5028.320738] LustreError: 154442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5028.334487] LustreError: 154442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5031.297412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.475053] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5031.694578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5031.743216] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5035.446103] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5037.035781] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5037.055075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5037.677885] Lustre: lustre-MDT0000: Denying connection for new client 8c3cb0e7-e49a-45ce-b2c3-5f967cb232de (at 192.168.202.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5047.267809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5047.280744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5047.287493] Lustre: Skipped 1 previous similar message [ 5047.304572] Lustre: Skipped 1 previous similar message [ 5052.391477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5052.406729] Lustre: Skipped 1 previous similar message [ 5057.507020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5057.527353] Lustre: Skipped 1 previous similar message [ 5060.064507] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5060.238428] Lustre: server umount lustre-OST0000 complete [ 5063.586388] LustreError: 157102:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5063.595352] Lustre: 156467:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5063.609351] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5063.628212] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5063.640092] LustreError: 156467:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f1269cd8 x1866869359293696/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5063.651760] LustreError: 156467:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5063.667812] LustreError: 156467:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5063.675615] Lustre: lustre-MDT0000: Recovery over after 0:26, of 2 clients 0 recovered and 2 were evicted. [ 5063.947818] Lustre: server umount lustre-MDT0000 complete [ 5067.217909] LustreError: 153709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385493 with bad export cookie 6774311799569871793 [ 5067.224409] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5067.230698] LustreError: 153709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5067.232329] LustreError: 153038:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5069.681122] Lustre: server umount lustre-MDT0001 complete [ 5078.667057] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5081.271807] Key type lgssc unregistered [ 5081.602696] LNet: 157836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5081.614413] LNetError: 157836:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5082.677926] LNet: Removed LNI 192.168.202.145@tcp [ 5083.336483] Key type .llcrypt unregistered [ 5083.343316] Key type ._llcrypt unregistered [ 5107.345333] Key type ._llcrypt registered [ 5107.347251] Key type .llcrypt registered [ 5107.500991] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5108.775041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5108.833094] alg: No test for adler32 (adler32-zlib) [ 5110.185748] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 5110.477150] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5112.175245] Key type lgssc registered [ 5113.319800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5121.027177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5121.057856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.487047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5126.872203] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5132.768696] LustreError: 159232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5132.782706] LustreError: 159232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5135.272105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5135.759888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5139.866983] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5145.678811] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5149.665476] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5157.162386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5157.659929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5164.112127] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5167.934132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5169.633987] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5176.288544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5176.292519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5176.312800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5178.339964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5178.343218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5178.347890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5178.374163] Lustre: Skipped 1 previous similar message [ 5182.385625] Lustre: server umount lustre-OST0000 complete [ 5187.040984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5187.051271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5187.073444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5193.703252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5193.706330] Lustre: Skipped 3 previous similar messages [ 5198.832281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5198.857622] Lustre: Skipped 1 previous similar message [ 5200.351640] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5200.558275] Lustre: server umount lustre-MDT0000 complete [ 5203.946431] LustreError: 159232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5203.970552] LustreError: 159232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5204.989245] LustreError: 159233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5205.011290] LustreError: 159233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5205.040133] LustreError: 161672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385631 with bad export cookie 17893414309161654930 [ 5205.046827] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5205.064131] LustreError: 161672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5205.547728] Lustre: server umount lustre-MDT0001 complete [ 5217.899277] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5221.143154] Key type lgssc unregistered [ 5221.407719] LNet: 162408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5221.418401] LNetError: 162408:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5221.437395] LNet: Removed LNI 192.168.202.145@tcp [ 5222.243566] Key type .llcrypt unregistered [ 5222.246150] Key type ._llcrypt unregistered [ 5258.093941] Key type ._llcrypt registered [ 5258.103080] Key type .llcrypt registered [ 5258.257839] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5259.732816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5259.916778] alg: No test for adler32 (adler32-zlib) [ 5261.049399] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 5261.287226] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5263.008168] Key type lgssc registered [ 5264.259500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5272.004537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5272.053410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.629246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5278.585046] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5283.822214] LustreError: 164097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5283.851592] LustreError: 164097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5287.234883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5287.500243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5291.232981] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5296.210034] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5299.576826] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5306.831371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5307.294754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5313.287846] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5318.585272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5320.258834] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5327.841642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5327.870658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5327.895850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5328.884426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5328.898944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5328.909326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5333.279820] Lustre: server umount lustre-OST0000 complete [ 5338.593550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5338.601275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5338.621741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5338.633284] Lustre: Skipped 1 previous similar message [ 5343.899285] Lustre: server umount lustre-MDT0000 complete [ 5344.225069] LustreError: 164096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5344.253581] LustreError: 164096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5347.968897] LustreError: 164078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385774 with bad export cookie 12628906255565806157 [ 5347.983442] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5347.987126] LustreError: 164078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5348.430158] Lustre: server umount lustre-MDT0001 complete [ 5358.328843] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5360.868871] Key type lgssc unregistered [ 5361.101771] LNet: 167271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5361.109870] LNetError: 167271:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5362.154938] LNet: Removed LNI 192.168.202.145@tcp [ 5362.840385] Key type .llcrypt unregistered [ 5362.844565] Key type ._llcrypt unregistered [ 5380.949676] Key type ._llcrypt registered [ 5380.953260] Key type .llcrypt registered [ 5381.054136] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5381.984659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5382.046138] alg: No test for adler32 (adler32-zlib) [ 5383.158242] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 5383.452137] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5385.120213] Key type lgssc registered [ 5386.255345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5395.224705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5395.239513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5396.538207] Lustre: *** cfs_fail_loc=135, val=0*** [ 5396.626272] LustreError: 168757:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5396.632641] LustreError: 168757:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.145@tcp: cfg command failed: rc = -2 [ 5396.641911] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5396.641911] [ 5396.648366] LustreError: MGC192.168.202.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5396.667552] LustreError: 168718:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5396.672484] LustreError: 168718:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5396.676602] LustreError: 168718:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5396.681527] LustreError: 168718:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5396.739244] Lustre: server umount lustre-MDT0000 complete [ 5396.741768] LustreError: 168718:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5402.120443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5402.497366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5405.863833] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5412.839723] LustreError: 169128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5412.851987] LustreError: 169128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5412.914485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5413.230430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5416.744437] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5420.627315] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5423.526148] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5427.699466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.725913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5427.730252] Lustre: Skipped 1 previous similar message [ 5431.408406] Lustre: server umount lustre-MDT0000 complete [ 5432.803672] LustreError: 169128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5432.830963] LustreError: 169128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5434.563193] LustreError: 169110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385861 with bad export cookie 16383560422149277357 [ 5434.575196] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5434.577889] LustreError: 169110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5434.870856] Lustre: server umount lustre-MDT0001 complete [ 5443.954279] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5446.507040] Key type lgssc unregistered [ 5446.774861] LNet: 171204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5446.783686] LNetError: 171204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5446.805387] LNet: Removed LNI 192.168.202.145@tcp [ 5447.650467] Key type .llcrypt unregistered [ 5447.652466] Key type ._llcrypt unregistered [ 5466.422263] Key type ._llcrypt registered [ 5466.426334] Key type .llcrypt registered [ 5466.509758] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5466.747331] Key type .llcrypt unregistered [ 5466.752756] Key type ._llcrypt unregistered [ 5478.954668] Key type ._llcrypt registered [ 5478.957566] Key type .llcrypt registered [ 5479.071779] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5479.710260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5479.892364] alg: No test for adler32 (adler32-zlib) [ 5480.897517] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 5481.094323] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5482.703177] Key type lgssc registered [ 5483.665180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5489.520614] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5489.534951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5505.055298] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5510.176847] LustreError: 173280:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 5536.863183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5542.328476] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5547.280350] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5554.215556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5554.543868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5554.595228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5557.818419] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5559.786694] LustreError: 173306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5559.803883] LustreError: 173306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5564.897611] LustreError: 173305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5565.149113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5565.387848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5568.921377] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5573.747538] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5576.790755] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5579.021648] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5581.799585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.808179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5581.815423] Lustre: Skipped 1 previous similar message [ 5581.825434] Lustre: Skipped 1 previous similar message [ 5586.925036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5586.933265] Lustre: Skipped 1 previous similar message [ 5587.404411] Lustre: server umount lustre-OST0000 complete [ 5591.010389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5591.021285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.054439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5592.034993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.158353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5597.171554] Lustre: Skipped 3 previous similar messages [ 5602.275599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5602.278127] Lustre: Skipped 1 previous similar message [ 5605.343096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5605.653955] Lustre: server umount lustre-MDT0000 complete [ 5607.394145] LustreError: 175440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.422141] LustreError: 175440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5609.542738] LustreError: 175603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386036 with bad export cookie 1379128579652461634 [ 5609.546938] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5609.555899] LustreError: 175603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5609.998077] Lustre: server umount lustre-MDT0001 complete [ 5619.931987] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5622.331597] Key type lgssc unregistered [ 5622.632755] LNet: 176697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5622.639601] LNetError: 176697:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5623.654276] LNet: Removed LNI 192.168.202.145@tcp [ 5624.396237] Key type .llcrypt unregistered [ 5624.398265] Key type ._llcrypt unregistered [ 5644.404501] Key type ._llcrypt registered [ 5644.408258] Key type .llcrypt registered [ 5644.493497] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5645.435834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5645.460449] alg: No test for adler32 (adler32-zlib) [ 5646.490209] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 5646.678755] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5648.335331] Key type lgssc registered [ 5649.370250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5656.186760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5656.205675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5657.631537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5661.580594] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5667.809330] LustreError: 178072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.819899] LustreError: 178072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5669.196625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5669.524444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5673.374969] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5677.754955] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5680.907112] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5686.344830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5686.616946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5691.469120] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5697.466568] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5703.038585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5706.160065] Lustre: Failing over lustre-MDT0000 [ 5706.462233] Lustre: server umount lustre-MDT0000 complete [ 5708.267650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5708.283386] LustreError: 178066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5708.294450] Lustre: Skipped 2 previous similar messages [ 5708.334073] LustreError: 178066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5713.377480] LustreError: 178072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5713.392256] LustreError: 178072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5718.498562] LustreError: 178782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5718.514162] LustreError: 178782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5723.615209] Lustre: 177384:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780386134/real 1780386134] req@000000004705f947 x1866870106905856/t0(0) o400->MGC192.168.202.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780386150 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5723.630836] LustreError: 178066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5723.641369] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5723.671382] LustreError: 178066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5725.108469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5733.860766] LustreError: 178068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5733.885224] LustreError: 178068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5734.175525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.213630] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5735.076868] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5737.169568] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5739.511865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5739.522508] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5739.559503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5740.218796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5743.856311] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5749.732623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5749.737492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5749.751579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5754.849968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5754.857436] Lustre: Skipped 1 previous similar message [ 5755.377679] Lustre: server umount lustre-OST0000 complete [ 5759.972566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.975184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5759.979917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5759.985364] Lustre: Skipped 2 previous similar messages [ 5759.988065] Lustre: Skipped 1 previous similar message [ 5765.089907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5765.104943] Lustre: Skipped 2 previous similar messages [ 5770.208333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5770.213809] Lustre: Skipped 1 previous similar message [ 5772.258210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5772.382038] Lustre: server umount lustre-MDT0000 complete [ 5774.597313] LustreError: 178052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386201 with bad export cookie 14130516248737877304 [ 5774.601307] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5774.607780] LustreError: 178052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5774.813991] Lustre: server umount lustre-MDT0001 complete [ 5782.440944] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5784.537790] Key type lgssc unregistered [ 5784.845636] LNet: 182481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5784.850237] LNetError: 182481:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5784.867174] LNet: Removed LNI 192.168.202.145@tcp [ 5785.533874] Key type .llcrypt unregistered [ 5785.537906] Key type ._llcrypt unregistered [ 5803.401962] Key type ._llcrypt registered [ 5803.404173] Key type .llcrypt registered [ 5803.480870] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5804.471608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5804.494132] alg: No test for adler32 (adler32-zlib) [ 5805.467849] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 5805.627893] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5807.263794] Key type lgssc registered [ 5808.220731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5814.015894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5814.036786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.486557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5818.460315] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5824.565958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5824.790726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5828.432201] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5832.678736] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5835.353583] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5840.802350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5841.065898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5842.103321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5844.859558] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5849.017303] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5851.355996] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5857.320774] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5867.807062] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5876.194945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5876.203520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5876.206862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5876.217764] Lustre: Skipped 1 previous similar message [ 5876.230666] Lustre: Skipped 1 previous similar message [ 5879.794957] Lustre: server umount lustre-OST0000 complete [ 5886.433214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5886.438242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5886.445648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5886.451311] Lustre: Skipped 1 previous similar message [ 5888.468500] Lustre: server umount lustre-MDT0000 complete [ 5890.753104] LustreError: 184575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386317 with bad export cookie 14809623068802135140 [ 5890.760703] LustreError: 184575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5890.762017] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5890.949787] Lustre: server umount lustre-MDT0001 complete [ 5897.360828] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5899.264784] Key type lgssc unregistered [ 5899.473193] LNet: 187343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5899.478467] LNetError: 187343:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5899.503337] LNet: Removed LNI 192.168.202.145@tcp [ 5900.036835] Key type .llcrypt unregistered [ 5900.048330] Key type ._llcrypt unregistered [ 5912.107944] Key type ._llcrypt registered [ 5912.110172] Key type .llcrypt registered [ 5912.200587] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5913.186538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5913.201436] alg: No test for adler32 (adler32-zlib) [ 5914.188823] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 5914.378298] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5916.047200] Key type lgssc registered [ 5916.819691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5921.838395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5921.859946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5923.155920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5925.974623] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5931.454197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.664602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5934.212844] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5937.836176] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5940.013366] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5944.231125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5944.443133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5947.507250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5948.066885] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5951.822509] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5956.305454] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5969.038107] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5973.789949] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5982.550390] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5992.933968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5992.935331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.939196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5992.956398] Lustre: Skipped 1 previous similar message [ 5996.638308] Lustre: server umount lustre-OST0000 complete [ 6003.169903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.178978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6003.183113] Lustre: Skipped 2 previous similar messages [ 6008.289434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6008.293177] Lustre: Skipped 1 previous similar message [ 6012.384411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6012.388243] Lustre: Skipped 1 previous similar message [ 6013.407881] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6013.508404] Lustre: server umount lustre-MDT0000 complete [ 6015.911572] LustreError: 188555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386442 with bad export cookie 7968836876651602885 [ 6015.912717] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6015.923167] LustreError: 188555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6016.101460] Lustre: server umount lustre-MDT0001 complete [ 6022.148617] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 6023.581965] Key type lgssc unregistered [ 6023.756503] LNet: 193328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6023.760688] LNetError: 193328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6023.777392] LNet: Removed LNI 192.168.202.145@tcp [ 6024.223416] Key type .llcrypt unregistered [ 6024.226708] Key type ._llcrypt unregistered [ 6037.462363] Key type ._llcrypt registered [ 6037.464264] Key type .llcrypt registered [ 6037.544266] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 6038.175483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6038.237666] alg: No test for adler32 (adler32-zlib) [ 6039.188900] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6039.329766] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 6040.983168] Key type lgssc registered [ 6041.573410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6046.268972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6046.280254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.529811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6049.895340] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6054.658040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6054.871202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6057.260799] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6060.382167] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6062.213938] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6066.374887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6066.673462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6070.370306] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6072.810435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6073.915918] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6083.040031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6083.043804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6083.053559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6085.089152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6085.097173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6085.100209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.946861] Lustre: server umount lustre-OST0000 complete [ 6090.210800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.215357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6090.222465] Lustre: Skipped 1 previous similar message [ 6090.230450] Lustre: Skipped 2 previous similar messages [ 6095.335483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6095.342343] Lustre: Skipped 1 previous similar message [ 6096.434068] Lustre: server umount lustre-MDT0000 complete [ 6098.788079] LustreError: 194681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386525 with bad export cookie 16700440545902886784 [ 6098.793414] LustreError: 194681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6098.795021] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6098.955130] Lustre: server umount lustre-MDT0001 complete [ 6104.446331] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 6106.041456] Key type lgssc unregistered [ 6106.245581] LNet: 198021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6106.248530] LNetError: 198021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6106.258404] LNet: Removed LNI 192.168.202.145@tcp [ 6106.738542] Key type .llcrypt unregistered [ 6106.740863] Key type ._llcrypt unregistered [ 6117.569665] Key type ._llcrypt registered [ 6117.575521] Key type .llcrypt registered [ 6117.654534] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 6118.404464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6118.422107] alg: No test for adler32 (adler32-zlib) [ 6119.372894] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6119.520559] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 6121.143135] Key type lgssc registered [ 6121.787637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6125.791829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6125.802290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6127.075940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6129.332887] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6134.476882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6134.710769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6137.110170] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6140.698219] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6142.901024] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6146.684583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6146.895249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6150.164753] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6152.045164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6153.840165] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6165.485484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6165.494187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6165.501573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6168.740789] Lustre: server umount lustre-OST0000 complete [ 6175.713356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6175.713668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6175.716091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6175.716096] Lustre: Skipped 1 previous similar message [ 6175.730678] Lustre: Skipped 2 previous similar messages [ 6177.037627] Lustre: server umount lustre-MDT0000 complete [ 6179.348525] LustreError: 200866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386606 with bad export cookie 9907485399218856976 [ 6179.353371] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6179.365765] LustreError: 200866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6179.683841] Lustre: server umount lustre-MDT0001 complete [ 6186.046170] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 6187.907418] Key type lgssc unregistered [ 6188.090929] LNet: 202568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6188.096988] LNetError: 202568:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6188.113185] LNet: Removed LNI 192.168.202.145@tcp [ 6188.759694] Key type .llcrypt unregistered [ 6188.761245] Key type ._llcrypt unregistered [ 6200.655918] Key type ._llcrypt registered [ 6200.658165] Key type .llcrypt registered [ 6200.728526] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 6201.212799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6201.224227] alg: No test for adler32 (adler32-zlib) [ 6202.191182] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6202.346817] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 6203.959175] Key type lgssc registered [ 6204.697968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6209.582432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6209.600394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.972433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6213.653915] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6218.463807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6218.646889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6220.657229] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6223.591932] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6225.502465] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6229.209430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6229.482698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6232.710138] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6235.627519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6235.948304] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6240.976052] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.026122] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6241.163478] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6244.433384] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6247.859655] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6248.850928] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6259.869882] Lustre: Permanently deactivating lustre-OST0001 [ 6259.879580] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6268.949025] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6271.725539] Lustre: Permanently reactivating lustre-OST0001 [ 6271.739669] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6271.743226] Lustre: Skipped 2 previous similar messages [ 6274.450456] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6274.459843] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6274.463658] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6274.468267] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6274.487652] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6284.768196] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6284.769682] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6284.774953] LustreError: Skipped 1 previous similar message [ 6284.776197] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6284.790773] Lustre: Skipped 2 previous similar messages [ 6286.323622] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6286.326884] Lustre: Skipped 2 previous similar messages [ 6288.625478] Lustre: server umount lustre-OST0001 complete [ 6291.430166] LustreError: 206713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6291.430809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.431552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6291.454501] LustreError: 206713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6291.484868] Lustre: Skipped 1 previous similar message [ 6296.548626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6296.549552] LustreError: 206713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6296.556049] Lustre: Skipped 2 previous similar messages [ 6296.568113] LustreError: 206713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6297.293616] Lustre: server umount lustre-OST0000 complete [ 6300.640048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6300.648086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6300.661126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6305.929134] Lustre: server umount lustre-MDT0000 complete [ 6306.786500] LustreError: 206260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6306.801592] LustreError: 206260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6308.190306] LustreError: 203773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386734 with bad export cookie 16854303093960561890 [ 6308.196890] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6308.197573] LustreError: 203773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6308.414299] Lustre: server umount lustre-MDT0001 complete [ 6314.448951] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 6316.158773] Key type lgssc unregistered [ 6316.331547] LNet: 208370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6316.340123] LNetError: 208370:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6316.358589] LNet: Removed LNI 192.168.202.145@tcp [ 6316.903448] Key type .llcrypt unregistered [ 6316.904546] Key type ._llcrypt unregistered [ 6341.190615] Key type ._llcrypt registered [ 6341.193170] Key type .llcrypt registered [ 6341.257292] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 6342.024623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6342.039609] alg: No test for adler32 (adler32-zlib) [ 6343.016605] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6343.179252] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 6344.783207] Key type lgssc registered [ 6345.507189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6349.967281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6349.985989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.175612] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6351.190395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6351.279204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6353.854485] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6358.889668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6358.976316] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6358.996229] Lustre: 210553:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6359.172027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6361.615946] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6364.705256] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6366.661816] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6370.736424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6370.854229] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6370.963249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6374.207167] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6377.342206] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6379.069514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6385.359420] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6393.435493] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6403.625591] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6410.694982] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6420.816507] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6442.485266] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6442.490480] Lustre: Skipped 2 previous similar messages [ 6483.127413] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6483.131150] Lustre: Skipped 4 previous similar messages [ 6551.154862] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6551.160264] Lustre: Skipped 8 previous similar messages [ 6553.058256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.063333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6553.063915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6553.068688] Lustre: Skipped 1 previous similar message [ 6558.178128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6558.181308] Lustre: Skipped 1 previous similar message [ 6559.196209] Lustre: server umount lustre-OST0000 complete [ 6563.297806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6563.299375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6563.303776] Lustre: Skipped 2 previous similar messages [ 6566.766295] Lustre: server umount lustre-MDT0000 complete [ 6568.315235] LustreError: 212838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386995 with bad export cookie 11498011525552866319 [ 6568.321447] LustreError: 212838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6568.321771] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6568.451820] Lustre: server umount lustre-MDT0001 complete [ 6572.372275] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 6573.551854] Key type lgssc unregistered [ 6573.688418] LNet: 215003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6573.691896] LNetError: 215003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6573.701496] LNet: Removed LNI 192.168.202.145@tcp [ 6574.037394] Key type .llcrypt unregistered [ 6574.039246] Key type ._llcrypt unregistered [ 6584.231424] Key type ._llcrypt registered [ 6584.232956] Key type .llcrypt registered [ 6584.279327] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 6584.618068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6584.626279] alg: No test for adler32 (adler32-zlib) [ 6585.500656] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6585.593425] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 6587.191101] Key type lgssc registered [ 6587.587259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6591.012034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6591.019496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6592.272169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6593.988915] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6597.626095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6597.776645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6599.357892] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6601.630410] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6603.143122] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6605.968596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6606.122476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6608.409992] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6610.584132] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6616.359176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6631.905199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6631.907551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6631.917032] Lustre: Skipped 1 previous similar message [ 6637.026704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6637.029341] Lustre: Skipped 2 previous similar messages [ 6637.463881] Lustre: server umount lustre-OST0000 complete [ 6639.074075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6639.078371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6639.085568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6642.146255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6642.147339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6642.153102] Lustre: Skipped 1 previous similar message [ 6645.174420] Lustre: server umount lustre-MDT0000 complete [ 6646.769060] LustreError: 216356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387073 with bad export cookie 7811723491919415418 [ 6646.769823] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.772987] LustreError: 216356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6646.913876] Lustre: server umount lustre-MDT0001 complete [ 6651.422867] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 6652.705040] Key type lgssc unregistered [ 6652.859801] LNet: 219725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6652.862996] LNetError: 219725:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6652.874350] LNet: Removed LNI 192.168.202.145@tcp [ 6653.247680] Key type .llcrypt unregistered [ 6653.249496] Key type ._llcrypt unregistered [ 6663.280081] Key type ._llcrypt registered [ 6663.281683] Key type .llcrypt registered [ 6663.329260] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 6663.737347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6663.779521] alg: No test for adler32 (adler32-zlib) [ 6664.679649] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6664.816040] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 6666.431189] Key type lgssc registered [ 6666.896220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6933.724798] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 6934.872245] Key type lgssc unregistered [ 6935.002459] LNet: 221700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6935.005624] LNetError: 221700:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6935.013308] LNet: Removed LNI 192.168.202.145@tcp [ 6935.319596] Key type .llcrypt unregistered [ 6935.320814] Key type ._llcrypt unregistered [ 6944.767602] Key type ._llcrypt registered [ 6944.768831] Key type .llcrypt registered [ 6944.816894] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 6952.423448] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 6952.919883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6952.929994] alg: No test for adler32 (adler32-zlib) [ 6953.815178] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6953.919377] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 6955.528435] Key type lgssc registered [ 6955.997875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6969.966146] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6685 sec ========== 04:03:16 (1780387396) [ 6970.734965] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:03:17 (1780387397) === [ 6972.308599] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:03:18 (1780387398) === [ 6989.503185] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 6990.679289] Key type lgssc unregistered [ 6990.846519] LNet: 226857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6990.850848] LNetError: 226857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6990.862535] LNet: Removed LNI 192.168.202.145@tcp [ 6991.231474] Key type .llcrypt unregistered [ 6991.235025] Key type ._llcrypt unregistered