[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490150985 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002349] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007365] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011142] LSM: Security Framework initializing [ 0.012062] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014081] *** VALIDATE selinux *** [ 0.022765] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027200] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030031] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031117] *** VALIDATE tmpfs *** [ 0.032496] *** VALIDATE proc *** [ 0.034218] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037187] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039112] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041028] Spectre V2 : User space: Vulnerable [ 0.042008] Speculative Store Bypass: Vulnerable [ 0.045103] debug: unmapping init [mem 0xffffffffb8859000-0xffffffffb8860fff] [ 0.047133] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.050011] ... bit width: 48 [ 0.051010] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053012] ... max period: 00007fffffffffff [ 0.054013] ... fixed-purpose events: 3 [ 0.055010] ... event mask: 000000070000000f [ 0.056311] rcu: Hierarchical SRCU implementation. [ 0.058523] smp: Bringing up secondary CPUs ... [ 0.059611] x86: Booting SMP configuration: [ 0.060023] .... node #0, CPUs: #1 #2 #3 [ 0.063427] smp: Brought up 1 node, 4 CPUs [ 0.065012] smpboot: Max logical packages: 1 [ 0.066016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150100] node 0 deferred pages initialised in 81ms [ 0.152210] devtmpfs: initialized [ 0.153228] x86/mm: Memory block size: 128MB [ 0.155721] gcov: version magic: 0x41383552 [ 0.157221] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159138] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162477] pinctrl core: initialized pinctrl subsystem [ 0.166196] [ 0.166768] ************************************************************* [ 0.167013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168014] ** ** [ 0.169020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170014] ** ** [ 0.171015] ** This means that this kernel is built to expose internal ** [ 0.172011] ** IOMMU data structures, which may compromise security on ** [ 0.173011] ** your system. ** [ 0.174014] ** ** [ 0.175014] ** If you see this message and you are not debugging the ** [ 0.176013] ** kernel, report this immediately to your vendor! ** [ 0.177014] ** ** [ 0.178012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179012] ************************************************************* [ 0.180900] NET: Registered protocol family 16 [ 0.181438] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.185014] cpuidle: using governor menu [ 0.186735] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.189563] PCI: Using configuration type 1 for base access [ 0.192130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202070] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.203019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.205066] cryptd: max_cpu_qlen set to 1000 [ 0.207114] ACPI: Added _OSI(Module Device) [ 0.208013] ACPI: Added _OSI(Processor Device) [ 0.209023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214328] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218463] ACPI: Interpreter enabled [ 0.219051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.220013] ACPI: Using IOAPIC for interrupt routing [ 0.221097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222454] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232018] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.233043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.234025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.235089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.237343] acpiphp: Slot [2] registered [ 0.238128] acpiphp: Slot [5] registered [ 0.239116] acpiphp: Slot [6] registered [ 0.240116] acpiphp: Slot [7] registered [ 0.241142] acpiphp: Slot [8] registered [ 0.242114] acpiphp: Slot [9] registered [ 0.243124] acpiphp: Slot [10] registered [ 0.244101] acpiphp: Slot [3] registered [ 0.245103] acpiphp: Slot [4] registered [ 0.246096] acpiphp: Slot [11] registered [ 0.247131] acpiphp: Slot [12] registered [ 0.248125] acpiphp: Slot [13] registered [ 0.249118] acpiphp: Slot [14] registered [ 0.250107] acpiphp: Slot [15] registered [ 0.251098] acpiphp: Slot [16] registered [ 0.252098] acpiphp: Slot [17] registered [ 0.253109] acpiphp: Slot [18] registered [ 0.254129] acpiphp: Slot [19] registered [ 0.255091] acpiphp: Slot [20] registered [ 0.256095] acpiphp: Slot [21] registered [ 0.257120] acpiphp: Slot [22] registered [ 0.258105] acpiphp: Slot [23] registered [ 0.259112] acpiphp: Slot [24] registered [ 0.260109] acpiphp: Slot [25] registered [ 0.261130] acpiphp: Slot [26] registered [ 0.262112] acpiphp: Slot [27] registered [ 0.263097] acpiphp: Slot [28] registered [ 0.264110] acpiphp: Slot [29] registered [ 0.265098] acpiphp: Slot [30] registered [ 0.266118] acpiphp: Slot [31] registered [ 0.267076] PCI host bridge to bus 0000:00 [ 0.268037] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.269026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.270053] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.271027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.272028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.273031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.274204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.276000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.278320] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.284016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.287052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.288021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.289022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.290020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.291574] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.292824] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.293050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.294889] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.297018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.306020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.308722] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.311855] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.314016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.317016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.324021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.329997] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.339017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.345016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.384032] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.401090] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.417024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.430035] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.452019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.472565] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.479023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.486022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.503024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.513727] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.522024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.529026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.546025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.556000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.562024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.566018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.577026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.594303] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.597482] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.604599] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.607758] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.610400] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.616167] iommu: Default domain type: Passthrough [ 0.618428] SCSI subsystem initialized [ 0.619204] ACPI: bus type USB registered [ 0.621138] usbcore: registered new interface driver usbfs [ 0.624150] usbcore: registered new interface driver hub [ 0.626106] usbcore: registered new device driver usb [ 0.628316] pps_core: LinuxPPS API ver. 1 registered [ 0.631018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.635095] PTP clock support registered [ 0.637213] EDAC MC: Ver: 3.0.0 [ 0.639548] PCI: Using ACPI for IRQ routing [ 0.642057] NetLabel: Initializing [ 0.644020] NetLabel: domain hash size = 128 [ 0.646021] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.648130] NetLabel: unlabeled traffic allowed by default [ 0.651167] vgaarb: loaded [ 0.652272] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.653013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.659841] clocksource: Switched to clocksource kvm-clock [ 0.775573] VFS: Disk quotas dquot_6.6.0 [ 0.777148] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.779528] *** VALIDATE ramfs *** [ 0.780489] *** VALIDATE hugetlbfs *** [ 0.781938] pnp: PnP ACPI init [ 0.785254] pnp: PnP ACPI: found 6 devices [ 0.804066] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.807545] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.809743] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.812124] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.814555] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.817058] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.820302] NET: Registered protocol family 2 [ 0.822900] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.827825] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.831531] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.837417] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.841377] TCP: Hash tables configured (established 65536 bind 65536) [ 0.844572] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.848173] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.851041] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.854566] NET: Registered protocol family 1 [ 0.859111] RPC: Registered named UNIX socket transport module. [ 0.861917] RPC: Registered udp transport module. [ 0.863814] RPC: Registered tcp transport module. [ 0.865593] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.868231] NET: Registered protocol family 44 [ 0.870110] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.872641] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.875066] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.877723] PCI: CLS 0 bytes, default 64 [ 0.879572] Unpacking initramfs... [ 2.355542] debug: unmapping init [mem 0xffff9ed47cc54000-0xffff9ed47ffbffff] [ 2.362608] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.365160] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.369851] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.903811] Initialise system trusted keyrings [ 2.905755] Key type blacklist registered [ 2.908563] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.918663] zbud: loaded [ 2.922144] *** VALIDATE nfs *** [ 2.923633] *** VALIDATE nfs4 *** [ 2.925821] pstore: using deflate compression [ 2.929283] Platform Keyring initialized [ 3.053664] NET: Registered protocol family 38 [ 3.055665] Key type asymmetric registered [ 3.057376] Asymmetric key parser 'x509' registered [ 3.059365] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.062644] io scheduler mq-deadline registered [ 3.064732] io scheduler kyber registered [ 3.066741] io scheduler bfq registered [ 3.068486] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.071755] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.074944] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.078176] ACPI: Power Button [PWRF] [ 3.083895] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.092071] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.107838] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.115808] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.136742] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.166366] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.196908] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.204130] Non-volatile memory driver v1.3 [ 3.205316] Linux agpgart interface v0.103 [ 3.236944] virtio_blk virtio1: [vda] 150608 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.239992] vda: detected capacity change from 0 to 77111296 [ 3.262517] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.265615] vdb: detected capacity change from 0 to 1073741824 [ 3.283906] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.287773] vdc: detected capacity change from 0 to 2621440000 [ 3.302274] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.305199] vdd: detected capacity change from 0 to 2621440000 [ 3.320899] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.323786] vde: detected capacity change from 0 to 4294967296 [ 3.341984] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.345126] vdf: detected capacity change from 0 to 4294967296 [ 3.351497] libphy: Fixed MDIO Bus: probed [ 3.371507] usbcore: registered new interface driver usbserial_generic [ 3.374235] usbserial: USB Serial support registered for generic [ 3.376760] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.381336] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.383307] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.385889] mousedev: PS/2 mouse device common for all mice [ 3.389962] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.394510] rtc_cmos 00:05: RTC can wake from S4 [ 3.399194] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.400747] rtc_cmos 00:05: registered as rtc0 [ 3.405328] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.405561] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.408147] intel_pstate: CPU model not supported [ 3.415092] hid: raw HID events driver (C) Jiri Kosina [ 3.417464] usbcore: registered new interface driver usbhid [ 3.419579] usbhid: USB HID core driver [ 3.421058] drop_monitor: Initializing network drop monitor service [ 3.423328] Initializing XFRM netlink socket [ 3.425445] NET: Registered protocol family 10 [ 3.428224] Segment Routing with IPv6 [ 3.429985] NET: Registered protocol family 17 [ 3.432516] mpls_gso: MPLS GSO support [ 3.440270] RAS: Correctable Errors collector initialized. [ 3.441823] AVX version of gcm_enc/dec engaged. [ 3.442885] AES CTR mode by8 optimization enabled [ 3.526540] sched_clock: Marking stable (3526510646, 0)->(4556532149, -1030021503) [ 3.529583] registered taskstats version 1 [ 3.531675] Loading compiled-in X.509 certificates [ 3.533654] zswap: loaded using pool lzo/zbud [ 3.560249] Key type big_key registered [ 3.574022] Key type encrypted registered [ 3.576047] ima: No TPM chip found, activating TPM-bypass! [ 3.578444] ima: Allocated hash algorithm: sha1 [ 3.580484] ima: No architecture policies found [ 3.582158] evm: Initialising EVM extended attributes: [ 3.583982] evm: security.selinux [ 3.585503] evm: security.ima [ 3.586677] evm: security.capability [ 3.587986] evm: HMAC attrs: 0x1 [ 3.590316] rtc_cmos 00:05: setting system clock to 2026-10-02 22:15:01 UTC (1790979301) [ 3.595820] debug: unmapping init [mem 0xffffffffb9803000-0xffffffffb99fffff] [ 3.598834] debug: unmapping init [mem 0xffffffffb8582000-0xffffffffb8858fff] [ 3.608122] Write protecting the kernel read-only data: 28672k [ 3.611879] debug: unmapping init [mem 0xffffffffb6c03000-0xffffffffb6dfffff] [ 3.614454] debug: unmapping init [mem 0xffffffffb7514000-0xffffffffb75fffff] [ 3.651843] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.659700] systemd[1]: Detected virtualization kvm. [ 3.661550] systemd[1]: Detected architecture x86-64. [ 3.663413] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.688333] systemd[1]: No hostname configured. [ 3.689656] systemd[1]: Set hostname to . [ 3.691148] random: systemd: uninitialized urandom read (16 bytes read) [ 3.693569] systemd[1]: Initializing machine ID from random generator. [ 3.836818] random: systemd: uninitialized urandom read (16 bytes read) [ 3.839438] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.843546] random: systemd: uninitialized urandom read (16 bytes read) [ 3.846417] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.853264] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.504038] device-mapper: uevent: version 1.0.3 [ 4.506793] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.338685] virtio_net virtio0 ens2: renamed from eth0 [ 5.398614] random: fast init done [ 5.409037] scsi host0: ata_piix [ 5.444077] scsi host1: ata_piix [ 5.466085] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.468989] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.029744] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.094148] random: crng init done [ 10.095728] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.668204] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.923821] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.190220] SELinux: Disabled at runtime. [ 12.255644] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.265589] systemd[1]: Detected virtualization kvm. [ 12.267573] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.826051] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.829574] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.838803] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.845778] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.849827] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.858442] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.867439] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ 13.001389] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.399928] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.779894] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.839061] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.992328] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.006750] EDAC sbridge: Ver: 1.1.2 [ 15.790724] Key type dns_resolver registered [ 16.097314] NFS: Registering the id_resolver key type [ 16.099262] Key type id_resolver registered [ 16.101010] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg258-server login: [ 65.054349] hrtimer: interrupt took 2529410 ns [ 80.191366] libcfs: loading out-of-tree module taints kernel. [ 80.322267] Key type ._llcrypt registered [ 80.325113] Key type .llcrypt registered [ 80.521266] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_hostid [ 100.124369] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 102.683201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 102.705117] alg: No test for adler32 (adler32-zlib) [ 104.417640] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 105.727940] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 107.737547] Key type lgssc registered [ 110.085048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 134.560017] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 190.601929] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 207.283888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 207.322216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.669243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 208.735871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 208.863603] Lustre: lustre-MDT0000: new disk, initializing [ 209.006975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.040861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 213.829902] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.988340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.148520] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 231.183475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 231.191366] Lustre: Skipped 1 previous similar message [ 231.363772] Lustre: lustre-MDT0001: new disk, initializing [ 231.532716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.587694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 231.594488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 237.206431] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.828193] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 254.418918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.760555] Lustre: lustre-OST0000: new disk, initializing [ 254.766330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 254.773602] Lustre: 8459:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.845148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.712573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 255.725495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 255.806333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 261.258684] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.298450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 277.440747] Lustre: lustre-OST0001: new disk, initializing [ 277.444628] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 277.448756] Lustre: 9532:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 277.523652] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 285.889368] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.849989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 287.856948] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 287.956851] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 298.721251] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 307.937265] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 316.196699] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing check_logdir /tmp/testlogs/ [ 321.696419] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing yml_node [ 325.441658] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 327.722512] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 330.147829] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 331.912149] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 18:20:27 EDT 2026 [ 348.619710] Lustre: DEBUG MARKER: only running tests: 0-31 [ 350.079430] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 351.385504] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 369.634215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 369.652928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.691218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 370.153801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.646985] LustreError: 12320:0:(ldlm_resource.c:1227:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (ffff9ed4c4cdd200) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 375.806811] Lustre: server umount lustre-MDT0000 complete [ 380.385382] LustreError: 6533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.410091] LustreError: 6533:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 380.490354] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979678 with bad export cookie 17238781858891456963 [ 380.497977] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 380.511934] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 385.519483] LustreError: 6528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 385.519623] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 385.520532] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 385.520540] Lustre: Skipped 4 previous similar messages [ 385.551617] LustreError: 6528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 385.579109] Lustre: Skipped 3 previous similar messages [ 387.000540] Lustre: server umount lustre-MDT0001 complete [ 390.750185] Lustre: server umount lustre-OST0000 complete [ 394.526820] Lustre: server umount lustre-OST0001 complete [ 401.302655] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_hostid [ 412.877901] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 454.822718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 455.155569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 455.212884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 455.341250] Lustre: lustre-MDT0000: new disk, initializing [ 455.440271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.459580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 459.587542] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 470.692309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.888506] Lustre: 15134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 470.959044] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 470.963930] Lustre: Skipped 1 previous similar message [ 471.166273] Lustre: lustre-MDT0001: new disk, initializing [ 471.319409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 471.373729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 471.386871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 476.355673] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 486.156926] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 491.631989] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 497.862996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 498.262239] Lustre: lustre-OST0000: new disk, initializing [ 498.271489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 498.277752] Lustre: 17102:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 498.374243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.002646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 500.019640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 500.138453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 504.663912] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 514.281385] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 519.978831] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 520.224982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 524.650993] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 524.851285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 530.915877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.919494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 530.963251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 530.971794] Lustre: Skipped 2 previous similar messages [ 532.664874] Lustre: server umount lustre-OST0000 complete [ 538.080650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 538.092657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.104521] Lustre: Skipped 1 previous similar message [ 538.112835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 542.876836] Lustre: server umount lustre-MDT0000 complete [ 546.257588] LustreError: 15125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979844 with bad export cookie 17238781858891458496 [ 546.259805] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 546.266877] LustreError: 15125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 546.832662] Lustre: server umount lustre-MDT0001 complete [ 550.015526] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 18:24:06 (1790979846) [ 556.986380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.341543] LustreError: 19366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.441037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 562.363511] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 562.658699] LustreError: 19367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.776768] LustreError: 19366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.906394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.092952] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 587.835894] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 594.205827] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 602.098601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 610.435747] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 618.565554] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 627.693647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 627.700447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.706744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 627.706754] Lustre: Skipped 2 previous similar messages [ 627.733686] Lustre: Skipped 2 previous similar messages [ 631.031475] Lustre: server umount lustre-OST0000 complete [ 637.932718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 637.933359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 637.943244] Lustre: Skipped 2 previous similar messages [ 641.155184] Lustre: server umount lustre-MDT0000 complete [ 643.048147] LustreError: 21723:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.102845] LustreError: 21723:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 644.294686] LustreError: 19346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979942 with bad export cookie 17238781858891459378 [ 644.300161] LustreError: 19346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 644.304154] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.588079] Lustre: server umount lustre-MDT0001 complete [ 655.858226] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 659.316949] Key type lgssc unregistered [ 659.569820] LNet: 22901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.572989] LNetError: 22901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 659.593594] LNet: Removed LNI 192.168.202.158@tcp [ 660.523176] Key type .llcrypt unregistered [ 660.524891] Key type ._llcrypt unregistered [ 685.745334] Key type ._llcrypt registered [ 685.749429] Key type .llcrypt registered [ 685.836286] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 687.278395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 687.308655] alg: No test for adler32 (adler32-zlib) [ 688.446443] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 688.807884] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 690.545492] Key type lgssc registered [ 692.157667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 699.907982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 699.934962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.372744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 706.304513] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 711.656641] LustreError: 24329:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.683335] LustreError: 24329:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 715.188555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.553204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 720.124599] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 726.772624] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 732.264128] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 739.204276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 739.524674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.149978] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 746.751468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 753.365908] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 771.015786] Lustre: server umount lustre-OST0000 complete [ 777.187389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 777.203720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 777.241345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 782.818928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 782.824866] Lustre: Skipped 2 previous similar messages [ 784.981501] Lustre: server umount lustre-MDT0000 complete [ 787.871847] Lustre: 23630:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790980069/real 1790980069] req@00000000c8e66678 x1877978629829888/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790980085 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 787.910216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 787.926146] Lustre: Skipped 1 previous similar message [ 787.939567] LustreError: 24330:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 787.970297] LustreError: 24330:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 788.974983] Lustre: 23632:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790980070/real 1790980070] req@0000000090ef8cd0 x1877978629830656/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790980086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 789.194030] LustreError: 24310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980087 with bad export cookie 14730831037362636028 [ 789.202405] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 789.206934] LustreError: 24310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 789.572479] Lustre: server umount lustre-MDT0001 complete [ 800.576709] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 803.700774] Key type lgssc unregistered [ 804.025757] LNet: 28304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.041038] LNetError: 28304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.067203] LNet: Removed LNI 192.168.202.158@tcp [ 805.046666] Key type .llcrypt unregistered [ 805.049413] Key type ._llcrypt unregistered [ 836.086722] Key type ._llcrypt registered [ 836.088366] Key type .llcrypt registered [ 836.275543] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 838.267222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 838.288231] alg: No test for adler32 (adler32-zlib) [ 839.358879] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 839.610484] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 841.335480] Key type lgssc registered [ 842.150550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 850.381924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 850.416322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.166863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.109443] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 862.702234] LustreError: 29737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 862.737074] LustreError: 29737:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 866.359764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.678902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 871.536935] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 879.018309] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 884.695612] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 897.652852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 897.963320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 904.063369] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 910.859655] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 922.558117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 931.808358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 931.817182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.849454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 932.835902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.864118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 932.872640] Lustre: Skipped 1 previous similar message [ 934.577852] Lustre: server umount lustre-OST0000 complete [ 938.465825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 938.482790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.498949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.076130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.076770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.101910] Lustre: Skipped 1 previous similar message [ 944.722615] Lustre: server umount lustre-MDT0000 complete [ 948.195540] LustreError: 29737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.461964] LustreError: 29718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980247 with bad export cookie 8066485608602210761 [ 949.467105] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 949.482513] LustreError: 29718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 950.056375] Lustre: server umount lustre-MDT0001 complete [ 963.173422] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 965.927528] Key type lgssc unregistered [ 966.306775] LNet: 33641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.319752] LNetError: 33641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.343896] LNet: Removed LNI 192.168.202.158@tcp [ 967.050525] Key type .llcrypt unregistered [ 967.053531] Key type ._llcrypt unregistered [ 992.952997] Key type ._llcrypt registered [ 992.956099] Key type .llcrypt registered [ 993.082886] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 994.654026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 994.739423] alg: No test for adler32 (adler32-zlib) [ 995.756887] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 996.036668] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 997.783208] Key type lgssc registered [ 999.180256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1007.192360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1007.241340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.599774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1013.353731] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1018.853091] LustreError: 35067:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.866038] LustreError: 35067:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1021.737525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.042436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1026.155691] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1033.116299] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1038.175310] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1045.247100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1045.675527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.553236] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1053.898670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1059.081228] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1067.489528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1067.494950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1067.507368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1069.544972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1069.551151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1069.559334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.574574] Lustre: Skipped 1 previous similar message [ 1073.713973] Lustre: server umount lustre-OST0000 complete [ 1078.242259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1078.254148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.264689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.898429] LustreError: 35063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.926506] LustreError: 35063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1085.531114] Lustre: server umount lustre-MDT0000 complete [ 1089.490446] LustreError: 35047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980387 with bad export cookie 15623574977775451205 [ 1089.492588] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1089.508597] LustreError: 35047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1089.925157] Lustre: server umount lustre-MDT0001 complete [ 1100.683683] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 1103.104845] Key type lgssc unregistered [ 1103.316962] LNet: 38598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.320677] LNetError: 38598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1103.340482] LNet: Removed LNI 192.168.202.158@tcp [ 1104.376119] Key type .llcrypt unregistered [ 1104.379043] Key type ._llcrypt unregistered [ 1128.178639] Key type ._llcrypt registered [ 1128.180951] Key type .llcrypt registered [ 1128.283166] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 1129.350268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1129.379652] alg: No test for adler32 (adler32-zlib) [ 1130.604374] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 1130.848723] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 1132.656044] Key type lgssc registered [ 1133.988894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1142.255848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1142.303557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.971057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1148.183866] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1153.509947] LustreError: 40028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1153.530562] LustreError: 40028:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1156.440800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1156.871591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1161.212486] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1168.549908] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1173.921208] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1181.219436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1181.680596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.627815] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1188.887304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1197.491204] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1204.192424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1204.204881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.217264] Lustre: Skipped 1 previous similar message [ 1204.219523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1204.230368] Lustre: Skipped 1 previous similar message [ 1205.374500] Lustre: lustre-OST0000: Not available for connect from 192.168.202.58@tcp (stopping) [ 1209.327072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1209.335712] Lustre: Skipped 1 previous similar message [ 1210.409376] Lustre: server umount lustre-OST0000 complete [ 1233.899286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1233.913868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.923622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.916861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.592291] Lustre: server umount lustre-MDT0000 complete [ 1241.056748] LustreError: 40028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.084386] LustreError: 40028:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1242.053712] LustreError: 40009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980539 with bad export cookie 5566815316208206043 [ 1242.069176] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1242.069914] LustreError: 40009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1248.485923] Lustre: server umount lustre-MDT0001 complete [ 1260.184677] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 1263.021675] Key type lgssc unregistered [ 1263.323936] LNet: 43737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.333305] LNetError: 43737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.355056] LNet: Removed LNI 192.168.202.158@tcp [ 1264.206916] Key type .llcrypt unregistered [ 1264.208940] Key type ._llcrypt unregistered [ 1290.757785] Key type ._llcrypt registered [ 1290.760519] Key type .llcrypt registered [ 1290.880928] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 1292.119044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1292.178279] alg: No test for adler32 (adler32-zlib) [ 1293.214343] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 1293.366796] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 1295.007269] Key type lgssc registered [ 1296.226460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1304.259368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1304.278929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.896120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1310.681580] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1315.299901] LustreError: 45167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.331400] LustreError: 45167:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1319.530846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.992488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1324.746801] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1332.298544] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1338.364773] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1345.320499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1345.641987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.784405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1351.392747] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1358.881580] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1370.600470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.602348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1370.626289] Lustre: Skipped 2 previous similar messages [ 1370.637906] Lustre: Skipped 2 previous similar messages [ 1371.756434] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.58@tcp (stopping) [ 1375.285309] Lustre: server umount lustre-MDT0000 complete [ 1375.724260] LustreError: 45166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.752608] LustreError: 45166:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1376.863717] LustreError: 45162:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1378.775972] LustreError: 45148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980676 with bad export cookie 11534319350690452800 [ 1378.776751] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1378.796804] LustreError: 45148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1380.832762] LustreError: 45161:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1380.835291] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.874388] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1383.840356] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1383.857141] Lustre: Skipped 1 previous similar message [ 1385.315690] Lustre: server umount lustre-MDT0001 complete [ 1400.514919] Lustre: server umount lustre-OST0000 complete [ 1417.616396] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 1420.676505] Key type lgssc unregistered [ 1420.955863] LNet: 49025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.963448] LNetError: 49025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1421.006891] LNet: Removed LNI 192.168.202.158@tcp [ 1421.901945] Key type .llcrypt unregistered [ 1421.904271] Key type ._llcrypt unregistered [ 1445.894375] Key type ._llcrypt registered [ 1445.900301] Key type .llcrypt registered [ 1445.988523] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 1447.285443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1447.361119] alg: No test for adler32 (adler32-zlib) [ 1448.396392] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 1448.590863] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 1450.239714] Key type lgssc registered [ 1451.460427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1458.703353] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1458.727133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1474.207671] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1479.327333] LustreError: 50429:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 1505.952892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.316945] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1523.327844] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1688.428620] Lustre: server umount lustre-OST0000 complete [ 1705.190538] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 1708.062975] Key type lgssc unregistered [ 1708.301559] LNet: 52102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.306509] LNetError: 52102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1708.320609] LNet: Removed LNI 192.168.202.158@tcp [ 1709.041309] Key type .llcrypt unregistered [ 1709.043946] Key type ._llcrypt unregistered [ 1731.569609] Key type ._llcrypt registered [ 1731.571898] Key type .llcrypt registered [ 1731.662212] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 1732.915475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1732.952603] alg: No test for adler32 (adler32-zlib) [ 1733.966836] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 1734.193385] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 1735.855230] Key type lgssc registered [ 1737.383454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1745.318674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1745.348404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.874249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1751.617838] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1756.145547] LustreError: 53539:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1756.168306] LustreError: 53539:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1761.256784] LustreError: 53540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.335331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.606628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1767.534838] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1774.967780] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1781.749470] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1789.717389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1790.055129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.118772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1796.628353] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1804.256774] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1807.399768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1807.402342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1812.482290] LustreError: 55364:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1812.500284] LustreError: 55364:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1812.728295] Lustre: server umount lustre-OST0000 complete [ 1817.571461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.595848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1817.610625] Lustre: Skipped 2 previous similar messages [ 1822.689109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1822.698063] Lustre: Skipped 1 previous similar message [ 1822.958624] Lustre: server umount lustre-MDT0000 complete [ 1827.249129] LustreError: 53519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981125 with bad export cookie 14826823020484169799 [ 1827.253026] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.258955] LustreError: 53519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1827.630861] Lustre: server umount lustre-MDT0001 complete [ 1839.691524] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 1843.062425] Key type lgssc unregistered [ 1843.463581] LNet: 57073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.468113] LNetError: 57073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.521432] LNet: Removed LNI 192.168.202.158@tcp [ 1845.437574] Key type .llcrypt unregistered [ 1845.439961] Key type ._llcrypt unregistered [ 1869.164958] Key type ._llcrypt registered [ 1869.172152] Key type .llcrypt registered [ 1869.268656] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 1870.442842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1870.498860] alg: No test for adler32 (adler32-zlib) [ 1871.526938] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 1871.751428] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 1873.400655] Key type lgssc registered [ 1874.827616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1883.247395] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1883.262684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.783606] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1903.903420] LustreError: 58473:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 1930.655758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1937.689305] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1947.084152] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1956.938678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.779743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.904546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1962.993807] LustreError: 58498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1963.028275] LustreError: 58498:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1963.505680] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1966.066369] LustreError: 59512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.094243] LustreError: 59512:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1971.175603] LustreError: 59512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.205198] LustreError: 59512:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1972.596135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.181258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1977.930737] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 1985.700176] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1991.063732] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1993.710817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1993.712923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.729653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1993.739392] Lustre: Skipped 1 previous similar message [ 1998.827430] LustreError: 58488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.849949] LustreError: 58488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1999.097737] Lustre: server umount lustre-OST0000 complete [ 2023.530404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.532409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.558034] Lustre: Skipped 2 previous similar messages [ 2023.565536] Lustre: Skipped 1 previous similar message [ 2028.823829] Lustre: server umount lustre-MDT0000 complete [ 2029.029742] LustreError: 61087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.046147] LustreError: 61087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2032.467479] LustreError: 58479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981330 with bad export cookie 1877732196246158819 [ 2032.475647] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2032.484805] LustreError: 58479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2038.867888] Lustre: server umount lustre-MDT0001 complete [ 2048.374704] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2051.337744] Key type lgssc unregistered [ 2051.656396] LNet: 62217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.661702] LNetError: 62217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2051.698813] LNet: Removed LNI 192.168.202.158@tcp [ 2052.699443] Key type .llcrypt unregistered [ 2052.701664] Key type ._llcrypt unregistered [ 2074.266269] Key type ._llcrypt registered [ 2074.268459] Key type .llcrypt registered [ 2074.358225] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 2075.364547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2075.526087] alg: No test for adler32 (adler32-zlib) [ 2076.586722] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 2076.806225] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 2078.545337] Key type lgssc registered [ 2079.726832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2087.232075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2087.253513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.719850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.474500] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2098.147181] LustreError: 63628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.169748] LustreError: 63628:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2102.555318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2102.887404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2108.251462] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2116.932777] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2122.935782] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2131.829782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2132.249498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.302899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2140.017974] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2148.495523] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2168.290464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.308151] LustreError: 65452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.445497] Lustre: server umount lustre-OST0000 complete [ 2173.414931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.442256] Lustre: Skipped 1 previous similar message [ 2173.460642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2173.485607] Lustre: Skipped 1 previous similar message [ 2178.531087] LustreError: 65751:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.557111] LustreError: 65751:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2178.682429] Lustre: server umount lustre-MDT0000 complete [ 2182.342623] LustreError: 63608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981480 with bad export cookie 14659862057073158169 [ 2182.360274] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2182.365912] LustreError: 63608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2182.639803] Lustre: server umount lustre-MDT0001 complete [ 2193.342554] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2196.310710] Key type lgssc unregistered [ 2196.601522] LNet: 67185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.609849] LNetError: 67185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2196.638862] LNet: Removed LNI 192.168.202.158@tcp [ 2197.568399] Key type .llcrypt unregistered [ 2197.570077] Key type ._llcrypt unregistered [ 2229.353674] Key type ._llcrypt registered [ 2229.356077] Key type .llcrypt registered [ 2229.460468] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 2231.088748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2231.116786] alg: No test for adler32 (adler32-zlib) [ 2232.258736] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 2232.666913] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 2234.383195] Key type lgssc registered [ 2235.867969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2245.026054] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 18:52:20 (1790981540) [ 2251.815577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2251.870584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.698283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.060375] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2264.035406] LustreError: 68913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.072026] LustreError: 68913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2269.034493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.157326] LustreError: 68914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.528326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2274.042033] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2280.778960] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2286.735947] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2293.684359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2293.980427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2299.124198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2300.635531] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2308.924642] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2314.866827] Lustre: Failing over lustre-MDT0000 [ 2315.118855] Lustre: server umount lustre-MDT0000 complete [ 2315.749963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2315.759270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.783834] LustreError: 68914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.843559] LustreError: 68913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.846487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.869182] LustreError: 68913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2319.900332] Lustre: Skipped 1 previous similar message [ 2324.968535] LustreError: 69928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.986254] LustreError: 69928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2325.455052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.596606] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2325.848240] Lustre: *** cfs_fail_loc=135, val=0*** [ 2325.985382] LustreError: 72000:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2326.003976] LustreError: 72000:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.158@tcp: cfg command failed: rc = -2 [ 2326.022915] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2326.022915] [ 2326.032920] LustreError: MGC192.168.202.158@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2326.058861] LustreError: 71988:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2326.069823] LustreError: 71988:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2326.083201] LustreError: 71988:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2326.098549] LustreError: 71988:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2326.142690] Lustre: server umount lustre-MDT0000 complete [ 2326.153501] LustreError: 71988:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2332.601440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.718596] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2333.050614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.118085] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2336.344300] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2337.812881] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2338.287200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2338.338535] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2338.405446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2346.464831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2346.471910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.487212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2348.523320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2348.529259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.559452] Lustre: server umount lustre-OST0000 complete [ 2358.755873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2358.767930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2358.771439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2358.771449] Lustre: Skipped 1 previous similar message [ 2358.812357] Lustre: Skipped 1 previous similar message [ 2361.500316] Lustre: server umount lustre-MDT0000 complete [ 2363.876691] LustreError: 68914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.912087] LustreError: 68914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2365.064404] LustreError: 68893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981662 with bad export cookie 12631169960032635404 [ 2365.067980] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2365.075191] LustreError: 68893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2365.333955] Lustre: server umount lustre-MDT0001 complete [ 2377.203496] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2380.385183] Key type lgssc unregistered [ 2380.769485] LNet: 73778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.780517] LNetError: 73778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2380.798836] LNet: Removed LNI 192.168.202.158@tcp [ 2382.082072] Key type .llcrypt unregistered [ 2382.086218] Key type ._llcrypt unregistered [ 2408.317332] Key type ._llcrypt registered [ 2408.319581] Key type .llcrypt registered [ 2408.418876] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 2410.325688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2410.519439] alg: No test for adler32 (adler32-zlib) [ 2411.751252] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 2412.099776] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 2413.903647] Key type lgssc registered [ 2415.340706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2423.660766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2423.697146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.238665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.397185] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2435.556623] LustreError: 75206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.584231] LustreError: 75206:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2439.654439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.987624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2443.985415] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2450.352486] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2455.855039] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2462.731465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2463.038675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.230408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2470.343622] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2478.998363] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2486.428726] Lustre: Failing over lustre-MDT0000 [ 2486.913303] LustreError: 75202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.691878] Lustre: server umount lustre-MDT0000 complete [ 2490.872246] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.876608] LustreError: 75935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.880521] Lustre: Skipped 1 previous similar message [ 2490.911525] LustreError: 75935:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2495.982828] LustreError: 76657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.008294] LustreError: 76657:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2500.073853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.165621] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2500.294381] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2500.394461] LustreError: 78294:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2500.404329] LustreError: 78294:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.158@tcp: cfg command failed: rc = -9 [ 2500.415121] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2500.415121] [ 2500.434316] LustreError: MGC192.168.202.158@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2500.451487] LustreError: 78282:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2500.458705] LustreError: 78282:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2500.475636] LustreError: 78282:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2500.484311] LustreError: 78282:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2500.509530] Lustre: server umount lustre-MDT0000 complete [ 2500.512477] LustreError: 78282:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2501.099097] LustreError: 75207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.116301] LustreError: 75207:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2507.969528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.077755] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2508.094563] LustreError: 75198:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2508.399708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2508.470322] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2513.292598] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2513.378376] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2513.381079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2513.549666] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2513.641954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2523.624458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2523.628166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2523.639388] Lustre: Skipped 2 previous similar messages [ 2523.657804] Lustre: Skipped 1 previous similar message [ 2528.742109] LustreError: 78940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.777256] LustreError: 78940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2528.940112] Lustre: server umount lustre-OST0000 complete [ 2533.860978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2533.878287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2533.878347] Lustre: Skipped 1 previous similar message [ 2538.782429] Lustre: server umount lustre-MDT0000 complete [ 2542.755704] LustreError: 79325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981840 with bad export cookie 1613547298996981553 [ 2542.755981] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2542.764888] LustreError: 79325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2543.313544] Lustre: server umount lustre-MDT0001 complete [ 2555.980526] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2560.374269] Key type lgssc unregistered [ 2560.728745] LNet: 80076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2560.737431] LNetError: 80076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2560.752843] LNet: Removed LNI 192.168.202.158@tcp [ 2561.764061] Key type .llcrypt unregistered [ 2561.766597] Key type ._llcrypt unregistered [ 2574.383337] Key type ._llcrypt registered [ 2574.385023] Key type .llcrypt registered [ 2574.529322] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2574.881448] Key type .llcrypt unregistered [ 2574.886265] Key type ._llcrypt unregistered [ 2598.757846] Key type ._llcrypt registered [ 2598.761330] Key type .llcrypt registered [ 2598.878361] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 2599.938847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2600.211984] alg: No test for adler32 (adler32-zlib) [ 2601.276643] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 2601.521225] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 2603.191223] Key type lgssc registered [ 2604.374907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2612.167610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2612.200303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2613.722412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.407657] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2623.970061] LustreError: 81993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.981513] LustreError: 81993:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2626.158653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.676847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2631.304115] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2639.288900] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2644.930593] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2652.793478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.062325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2658.154443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2658.943654] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2666.026580] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2681.831737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2681.833867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.845595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2681.879114] Lustre: Skipped 1 previous similar message [ 2685.450804] Lustre: server umount lustre-OST0000 complete [ 2692.065643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.079827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2692.082761] Lustre: Skipped 1 previous similar message [ 2697.192381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2697.195627] Lustre: Skipped 1 previous similar message [ 2701.284738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2701.299141] Lustre: Skipped 1 previous similar message [ 2703.327126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2703.579584] Lustre: server umount lustre-MDT0000 complete [ 2706.414857] LustreError: 82720:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.450143] LustreError: 82720:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2707.774481] LustreError: 81972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982005 with bad export cookie 12905043642462127153 [ 2707.794453] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2708.154655] Lustre: server umount lustre-MDT0001 complete [ 2718.707570] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2720.988658] Key type lgssc unregistered [ 2721.277782] LNet: 85546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2721.289854] LNetError: 85546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2722.348130] LNet: Removed LNI 192.168.202.158@tcp [ 2723.501071] Key type .llcrypt unregistered [ 2723.502932] Key type ._llcrypt unregistered [ 2746.877235] Key type ._llcrypt registered [ 2746.879626] Key type .llcrypt registered [ 2746.990216] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 2747.953555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2747.995239] alg: No test for adler32 (adler32-zlib) [ 2749.211815] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 2749.448552] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 2751.111223] Key type lgssc registered [ 2752.356465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2760.822736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2760.881176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.318212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.181201] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2772.450573] LustreError: 86973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2772.493136] LustreError: 86973:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2775.208412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.573463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2779.587700] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2787.256378] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2792.941521] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2800.709560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2801.062433] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2801.269151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.883497] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2807.292137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2813.561940] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2821.619703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2821.623136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.643954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2822.635287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2822.640409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.681899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2822.692832] Lustre: Skipped 1 previous similar message [ 2824.815445] Lustre: server umount lustre-OST0000 complete [ 2831.840400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2831.849251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.895363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.038186] Lustre: server umount lustre-MDT0000 complete [ 2837.996130] LustreError: 86972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2838.021255] LustreError: 86972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2838.818556] LustreError: 86953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982136 with bad export cookie 1545546507089615039 [ 2838.830827] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2838.834304] LustreError: 86953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2839.186670] Lustre: server umount lustre-MDT0001 complete [ 2855.011656] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2857.336435] Key type lgssc unregistered [ 2857.592559] LNet: 90519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2857.598306] LNetError: 90519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2857.612210] LNet: Removed LNI 192.168.202.158@tcp [ 2858.561597] Key type .llcrypt unregistered [ 2858.567264] Key type ._llcrypt unregistered [ 2880.985833] Key type ._llcrypt registered [ 2880.988479] Key type .llcrypt registered [ 2881.092923] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 2881.832858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2881.915172] alg: No test for adler32 (adler32-zlib) [ 2883.000374] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 2883.197282] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 2884.856186] Key type lgssc registered [ 2885.873618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2893.721722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2893.739269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.202205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.617237] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2905.571384] LustreError: 91929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.596580] LustreError: 91929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2907.503389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.851359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2912.327434] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2919.547854] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2925.049688] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2931.449089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2931.686984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.859915] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 2940.923952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2945.116167] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2956.256791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2956.264901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.286858] Lustre: Skipped 1 previous similar message [ 2956.289265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2957.735645] Lustre: server umount lustre-OST0000 complete [ 2964.463515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2964.469137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.481654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.500472] Lustre: Skipped 1 previous similar message [ 2966.501894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2966.506610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2968.866920] Lustre: server umount lustre-MDT0000 complete [ 2971.620889] LustreError: 92657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.688587] LustreError: 92657:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2973.641666] LustreError: 91908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982271 with bad export cookie 4455158018248956646 [ 2973.642204] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2973.650175] LustreError: 91908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2974.136531] Lustre: server umount lustre-MDT0001 complete [ 2985.102951] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 2987.821742] Key type lgssc unregistered [ 2988.161598] LNet: 95459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2988.175864] LNetError: 95459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2988.195976] LNet: Removed LNI 192.168.202.158@tcp [ 2989.157246] Key type .llcrypt unregistered [ 2989.161609] Key type ._llcrypt unregistered [ 3011.126938] Key type ._llcrypt registered [ 3011.131633] Key type .llcrypt registered [ 3011.220503] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 3012.130043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3012.172097] alg: No test for adler32 (adler32-zlib) [ 3013.199711] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 3013.469318] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 3015.136315] Key type lgssc registered [ 3016.265430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3024.209828] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3024.246501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3039.775780] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3044.895763] LustreError: 96863:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 3071.519257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3077.814380] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3084.413390] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3089.378491] Lustre: server umount lustre-OST0000 complete [ 3096.392162] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 19:06:32 (1790982392) [ 3101.524156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.967094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3105.900901] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3107.296719] LustreError: 98501:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.313807] LustreError: 98501:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3111.397874] LustreError: 98500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.910994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.229538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3118.666151] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3127.006262] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3133.587023] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3140.749635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3141.151692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3146.300786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3150.420144] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3159.463798] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3171.809918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3171.814144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.848020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3174.884063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3174.892089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3174.892568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.936939] Lustre: server umount lustre-OST0000 complete [ 3181.024362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3181.034240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3181.057937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3181.077906] Lustre: Skipped 1 previous similar message [ 3185.120324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.122844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.739268] Lustre: server umount lustre-MDT0000 complete [ 3190.241613] LustreError: 98497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.265510] LustreError: 98497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3191.133255] LustreError: 98482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982489 with bad export cookie 6157530649484909321 [ 3191.152796] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3191.538108] Lustre: server umount lustre-MDT0001 complete [ 3203.412272] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 3206.684634] Key type lgssc unregistered [ 3207.180334] LNet: 102190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3207.201644] LNetError: 102190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3207.224865] LNet: Removed LNI 192.168.202.158@tcp [ 3208.485486] Key type .llcrypt unregistered [ 3208.488212] Key type ._llcrypt unregistered [ 3245.540372] Key type ._llcrypt registered [ 3245.542881] Key type .llcrypt registered [ 3245.700366] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 3246.491577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3246.551420] alg: No test for adler32 (adler32-zlib) [ 3247.830691] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 3248.081266] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 3249.775286] Key type lgssc registered [ 3250.894844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3258.208190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3258.239460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.768193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.071547] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3269.101336] LustreError: 104380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.133293] LustreError: 104380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3273.489383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.934847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3279.484392] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3288.720332] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3294.737380] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3303.541589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3304.142750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3305.154455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3312.120339] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3320.692500] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3331.042547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3331.050715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.080258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3334.125945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3334.132924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3334.136315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3334.181036] Lustre: Skipped 1 previous similar message [ 3337.023293] Lustre: server umount lustre-OST0000 complete [ 3344.354126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3344.374190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3344.386250] Lustre: Skipped 1 previous similar message [ 3344.412841] Lustre: Skipped 1 previous similar message [ 3348.366420] Lustre: server umount lustre-MDT0000 complete [ 3349.479321] LustreError: 105108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.501352] LustreError: 105108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3351.721443] LustreError: 104360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982649 with bad export cookie 10371246369668934761 [ 3351.742355] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3352.197551] Lustre: server umount lustre-MDT0001 complete [ 3363.967590] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 3366.885394] Key type lgssc unregistered [ 3367.145695] LNet: 107915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3367.153432] LNetError: 107915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3367.174618] LNet: Removed LNI 192.168.202.158@tcp [ 3368.113346] Key type .llcrypt unregistered [ 3368.118289] Key type ._llcrypt unregistered [ 3387.930328] Key type ._llcrypt registered [ 3387.933337] Key type .llcrypt registered [ 3388.016274] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 3388.945815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3389.024275] alg: No test for adler32 (adler32-zlib) [ 3390.178301] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 3390.432860] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 3392.151212] Key type lgssc registered [ 3393.626530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3403.072574] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3403.097429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3418.783464] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3423.903340] LustreError: 109173:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 3450.591717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3458.770551] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3467.908900] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3474.530856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.747353] LustreError: MGC192.168.202.158@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3474.760609] LustreError: 110198:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3474.765978] LustreError: 110198:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3474.784907] LustreError: 110198:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3474.850688] Lustre: server umount lustre-MDT0000 complete [ 3474.853632] LustreError: 110198:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3491.967518] Lustre: server umount lustre-OST0000 complete [ 3501.021116] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_hostid [ 3509.703361] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 3546.245141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.427151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3546.443871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3546.493470] Lustre: lustre-MDT0000: new disk, initializing [ 3546.540634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.558501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3550.319383] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3562.441819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.639308] Lustre: 112988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3562.815155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3562.842467] Lustre: Skipped 1 previous similar message [ 3562.991407] Lustre: lustre-MDT0001: new disk, initializing [ 3563.135989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3563.197155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3563.225739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3568.629958] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3579.613751] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3586.595108] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3593.242456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3593.476376] Lustre: lustre-OST0000: new disk, initializing [ 3593.479759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3593.485042] Lustre: 114959:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3593.605788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3600.299929] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3600.525659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3600.535822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3600.750847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3610.040767] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3614.868601] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3615.055610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3619.778330] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3620.062948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3626.472639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3626.473978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3626.485370] LustreError: Skipped 1 previous similar message [ 3626.491582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3626.542738] Lustre: Skipped 1 previous similar message [ 3628.646599] Lustre: server umount lustre-OST0000 complete [ 3634.659413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3634.675059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.707979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.710818] Lustre: Skipped 1 previous similar message [ 3636.705274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3636.705947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.407452] Lustre: server umount lustre-MDT0000 complete [ 3641.827692] LustreError: 112997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3641.867590] LustreError: 112997:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3642.545552] LustreError: 112980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982940 with bad export cookie 10055530700998962423 [ 3642.556071] LustreError: 112980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3642.556876] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3642.774932] Lustre: server umount lustre-MDT0001 complete [ 3651.599832] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 19:15:47 (1790982947) [ 3655.154688] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3674.620406] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_hostid [ 3685.152719] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 3720.864234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.076440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3721.092671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3721.174593] Lustre: lustre-MDT0000: new disk, initializing [ 3721.370688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3721.390992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3726.659647] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3740.319116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3740.498615] Lustre: 119861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3740.564242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3740.569109] Lustre: Skipped 1 previous similar message [ 3740.752246] Lustre: lustre-MDT0001: new disk, initializing [ 3740.945995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3741.001356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3741.014623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3747.496487] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3758.090180] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3764.299756] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3772.360787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3772.674579] Lustre: lustre-OST0000: new disk, initializing [ 3772.680529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3772.700382] Lustre: 121831:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3772.811197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3779.257735] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3781.670833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3781.682821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3781.805349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3789.161915] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3793.757050] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3794.065259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3799.625871] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3799.872943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3802.597886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.614278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.635321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.640829] Lustre: Skipped 1 previous similar message [ 3807.202574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3807.210971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3807.232586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3808.610535] Lustre: server umount lustre-OST0000 complete [ 3812.833620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3817.952884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3817.970223] Lustre: Skipped 4 previous similar messages [ 3819.174142] Lustre: server umount lustre-MDT0000 complete [ 3823.079716] LustreError: 119872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3823.099480] LustreError: 119872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3823.573567] LustreError: 119851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983121 with bad export cookie 10055530700998963305 [ 3823.580624] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3823.593863] LustreError: 119851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3824.310980] Lustre: server umount lustre-MDT0001 complete [ 3834.038276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3834.488513] LustreError: 124046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3834.618444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3838.717515] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3839.976097] LustreError: 124047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.093405] LustreError: 124046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.233656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.632752] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3861.609419] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3867.280602] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3874.328116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3881.303418] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 3889.036810] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3898.680186] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3903.456643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3903.462400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3903.475577] Lustre: Skipped 2 previous similar messages [ 3903.482185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3903.486779] Lustre: Skipped 1 previous similar message [ 3908.585120] LustreError: 125869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3908.595100] LustreError: 125869:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3908.830942] Lustre: server umount lustre-OST0000 complete [ 3919.883562] Lustre: server umount lustre-MDT0000 complete [ 3923.560193] LustreError: 126924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983221 with bad export cookie 10055530700998964180 [ 3923.561039] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3923.568539] LustreError: 126924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3923.790677] Lustre: server umount lustre-MDT0001 complete [ 3933.873076] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 3936.444520] Key type lgssc unregistered [ 3936.748928] LNet: 127676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3936.761951] LNetError: 127676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3936.798739] LNet: Removed LNI 192.168.202.158@tcp [ 3937.729727] Key type .llcrypt unregistered [ 3937.730936] Key type ._llcrypt unregistered [ 3956.774369] Key type ._llcrypt registered [ 3956.779444] Key type .llcrypt registered [ 3956.932434] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_hostid [ 3973.280982] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 3975.009856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3975.074096] alg: No test for adler32 (adler32-zlib) [ 3976.190795] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 3976.489295] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 3978.199254] Key type lgssc registered [ 3979.448509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4014.533855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4014.608262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4015.801631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4015.820860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4015.880369] Lustre: lustre-MDT0000: new disk, initializing [ 4015.938886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.952841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4020.660976] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4032.304749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4032.424330] Lustre: 130846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4032.477568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4032.480630] Lustre: Skipped 1 previous similar message [ 4032.571572] Lustre: lustre-MDT0001: new disk, initializing [ 4032.721714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4032.803222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4032.833912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4037.009685] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4047.339656] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4054.573429] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4061.993936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4062.216757] Lustre: lustre-OST0000: new disk, initializing [ 4062.219399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4062.224749] Lustre: 132813:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4062.288822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.852862] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4069.456081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4069.475536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4069.645766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4078.253529] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4082.882525] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4083.185939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4087.989894] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4088.214515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4090.336150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4090.338685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4090.342293] LustreError: Skipped 1 previous similar message [ 4090.355069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4090.363531] Lustre: Skipped 1 previous similar message [ 4090.381237] Lustre: Skipped 1 previous similar message [ 4095.462508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4095.467658] Lustre: Skipped 1 previous similar message [ 4096.727069] Lustre: server umount lustre-OST0000 complete [ 4100.585180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4100.587873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4100.609024] Lustre: Skipped 1 previous similar message [ 4100.636441] Lustre: Skipped 1 previous similar message [ 4105.701736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4105.711944] Lustre: Skipped 1 previous similar message [ 4107.037817] Lustre: server umount lustre-MDT0000 complete [ 4110.968493] LustreError: 130839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983408 with bad export cookie 18399474564084293887 [ 4110.980864] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4111.470992] Lustre: server umount lustre-MDT0001 complete [ 4120.827736] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 19:23:36 (1790983416) [ 4127.282381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.679320] LustreError: 135226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4127.697675] LustreError: 135226:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4127.753215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4132.722718] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4132.832253] LustreError: 135227:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.958421] LustreError: 135226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.750983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4142.217485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4147.592431] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4156.172248] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4162.306510] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4166.379493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4166.394604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4166.405167] Lustre: Skipped 1 previous similar message [ 4170.660443] Lustre: server umount lustre-MDT0000 complete [ 4171.748482] LustreError: 135226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.764387] LustreError: 135226:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4174.437896] LustreError: 135961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983472 with bad export cookie 18399474564084294769 [ 4174.453413] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4174.458940] LustreError: 135961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4175.010036] Lustre: server umount lustre-MDT0001 complete [ 4182.816141] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 19:24:38 (1790983478) [ 4188.990277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4204.575671] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4209.695498] LustreError: 137641:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 4235.487407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4243.178249] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4251.185743] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4253.128405] Lustre: server umount lustre-OST0000 complete [ 4261.455993] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 19:25:57 (1790983557) [ 4268.165952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4268.500480] LustreError: 139088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4268.588145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4272.922298] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4278.767831] LustreError: 139089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4278.781622] LustreError: 139089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4281.987243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.149434] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4294.947800] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4301.223650] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4309.070320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4309.529345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4309.537630] Lustre: Skipped 1 previous similar message [ 4318.164277] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4325.559588] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4338.149192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.158774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4338.160750] Lustre: Skipped 2 previous similar messages [ 4338.180948] Lustre: Skipped 2 previous similar messages [ 4339.718701] Lustre: server umount lustre-MDT0000 complete [ 4343.135076] LustreError: 139069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983641 with bad export cookie 18399474564084295434 [ 4343.143350] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4343.146850] LustreError: 139069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4343.273244] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4343.276515] LustreError: 139085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4343.300727] Lustre: Skipped 1 previous similar message [ 4343.593904] Lustre: server umount lustre-MDT0001 complete [ 4349.521669] Lustre: server umount lustre-OST0000 complete [ 4357.486182] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 19:27:33 (1790983653) [ 4365.559918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4370.910594] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4376.548111] LustreError: 142755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4376.571522] LustreError: 142755:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4378.931284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4379.165152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4379.174940] Lustre: Skipped 1 previous similar message [ 4383.543581] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4392.102533] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4397.617331] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4404.436237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4411.017022] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4411.387190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4419.417510] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4424.365590] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4424.566656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4430.151505] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4430.427497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4435.939082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4435.942674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4435.961408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4436.971356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4438.937834] Lustre: server umount lustre-OST0000 complete [ 4445.665091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4449.112409] Lustre: server umount lustre-MDT0000 complete [ 4452.323189] LustreError: 142754:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.342931] LustreError: 142754:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4453.084161] LustreError: 142737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983750 with bad export cookie 18399474564084296498 [ 4453.107665] LustreError: 142737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4453.133489] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4453.534464] Lustre: server umount lustre-MDT0001 complete [ 4463.011722] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 19:29:18 (1790983758) [ 4470.994183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4486.623940] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4491.743794] LustreError: 146913:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 4517.471261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4517.477117] Lustre: Skipped 1 previous similar message [ 4525.627321] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4534.496892] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4542.662401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4543.519197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4548.185437] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4557.844354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4562.399996] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4572.705794] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4581.623920] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4588.689900] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4589.135991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4594.306572] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4594.463249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4599.269334] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.278108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4599.280996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4599.281029] Lustre: Skipped 5 previous similar messages [ 4599.299565] Lustre: Skipped 4 previous similar messages [ 4602.338769] Lustre: server umount lustre-MDT0000 complete [ 4604.393517] LustreError: 146939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.423682] LustreError: 146939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4608.131506] LustreError: 146921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983906 with bad export cookie 18399474564084297527 [ 4608.144185] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4608.145651] LustreError: 146921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4608.358402] Lustre: server umount lustre-MDT0001 complete [ 4623.721498] Lustre: server umount lustre-OST0000 complete [ 4634.932359] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 19:32:10 (1790983930) [ 4644.171701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4659.937334] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4665.055628] LustreError: 151114:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 4697.387466] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4705.478728] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4712.274279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4712.902109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4717.694106] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4726.317958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.702621] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4740.639586] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4747.253375] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4754.184632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4754.360209] Lustre: lustre-OST0001: new disk, initializing [ 4754.363668] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4754.369180] Lustre: 153985:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4760.092805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4760.108616] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4760.220504] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4760.468670] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4771.292156] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4777.349311] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4777.552848] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4782.843892] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4783.036097] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4785.641961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4785.659579] Lustre: Skipped 1 previous similar message [ 4785.678438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4785.691247] Lustre: Skipped 2 previous similar messages [ 4791.302164] Lustre: server umount lustre-OST0000 complete [ 4801.583075] Lustre: server umount lustre-OST0001 complete [ 4808.673407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4812.803794] Lustre: server umount lustre-MDT0000 complete [ 4817.060136] LustreError: 151122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790984114 with bad export cookie 18399474564084298479 [ 4817.066950] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4817.565109] Lustre: server umount lustre-MDT0001 complete [ 4841.534326] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 19:35:37 (1790984137) [ 4843.198843] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4844.997087] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4846.926140] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 19:35:42 (1790984142) [ 4852.958724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4853.189739] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4853.216697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4853.372538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4853.375316] Lustre: Skipped 6 previous similar messages [ 4857.735323] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4866.273155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4866.370101] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4866.397747] Lustre: 157542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4871.462353] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4878.851712] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4884.549221] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4892.297359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4892.509974] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4899.171756] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4902.074719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4906.841375] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4912.175100] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4912.401102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4916.849729] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4917.058917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4922.338760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4922.351113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.367499] Lustre: Skipped 5 previous similar messages [ 4922.377212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4922.383255] Lustre: Skipped 10 previous similar messages [ 4933.089323] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4933.432851] Lustre: server umount lustre-OST0000 complete [ 4957.322962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4963.898563] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4963.935143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4963.938987] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4964.542959] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 4972.495382] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4978.151850] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4978.397228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4982.843166] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4983.007436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4989.074507] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9037d1d94000.ost_server_uuid 50 [ 4991.084736] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9037d1d94000.ost_server_uuid in FULL state after 0 sec [ 4991.415502] LustreError: 159416:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.58@tcp) returned error from blocking AST (req@00000000936c7886 x1877982077788544 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000006e73e8d7/0xff5810a1a6147c49 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.58@tcp remote: 0xef734233801c4c8c expref: 5 pid: 157551 timeout: 5091 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4991.448441] LustreError: lustre-MDT0000: A client on nid 192.168.202.58@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4991.459477] LustreError: 157540:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.58@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000006e73e8d7/0xff5810a1a6147c49 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.58@tcp remote: 0xef734233801c4c8c expref: 6 pid: 157551 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4997.090082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4997.101596] LustreError: Skipped 1 previous similar message [ 5002.823734] Lustre: server umount lustre-OST0000 complete [ 5012.776039] Lustre: server umount lustre-MDT0000 complete [ 5015.008307] LustreError: 157556:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5015.023486] LustreError: 157556:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 5017.113341] LustreError: 157535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790984315 with bad export cookie 18399474564084299662 [ 5017.117799] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5017.133782] LustreError: 157535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5023.982670] Lustre: server umount lustre-MDT0001 complete [ 5038.597687] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5041.613951] Key type lgssc unregistered [ 5041.864274] LNet: 163663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5041.868988] LNetError: 163663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5041.880945] LNet: Removed LNI 192.168.202.158@tcp [ 5043.011693] Key type .llcrypt unregistered [ 5043.015950] Key type ._llcrypt unregistered [ 5069.938373] Key type ._llcrypt registered [ 5069.941062] Key type .llcrypt registered [ 5070.075203] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 5070.962212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5071.034855] alg: No test for adler32 (adler32-zlib) [ 5072.124243] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 5072.350543] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 5074.033262] Key type lgssc registered [ 5075.434532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5084.178943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5084.205667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5085.669746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5090.174661] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5095.909496] LustreError: 165091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5095.937748] LustreError: 165091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5098.699508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5099.058397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5103.791070] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5111.334376] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5117.130319] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5123.799520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5124.195449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5130.219942] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5130.398171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5138.139748] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5147.475231] Lustre: Failing over lustre-MDT0000 [ 5148.030217] Lustre: server umount lustre-MDT0000 complete [ 5149.158098] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.167496] Lustre: Skipped 1 previous similar message [ 5149.167531] LustreError: 165819:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5150.346771] LustreError: 165826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5150.378101] LustreError: 165826:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5154.276975] LustreError: 165086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5154.322932] LustreError: 165086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5159.401231] LustreError: 165085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5159.434678] LustreError: 165085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5160.866853] LustreError: 165071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.58@tcp arrived at 1790984458 with bad export cookie 4753392659385584734 [ 5160.901472] LustreError: 165071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5165.537196] Lustre: 164390:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790984447/real 1790984447] req@000000008674c5e2 x1877983226785152/t0(0) o400->MGC192.168.202.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790984463 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5165.580372] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5169.328855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5170.675814] LustreError: 165091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5170.704751] LustreError: 165091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5176.845239] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x41f771706ca08cc7 [ 5176.859540] LustreError: 168151:0:(ldlm_resource.c:1227:ldlm_resource_complain()) MGC192.168.202.158@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000004c9a2cdc) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 5176.901698] Lustre: MGC192.168.202.158@tcp: Connection restored to 0@lo (at 0@lo) [ 5177.260051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5177.338243] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5182.203947] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5182.439622] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5182.456182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5185.048519] Lustre: lustre-MDT0000: Denying connection for new client ca8d68c0-0ac3-4799-abfb-0d2b219bdb22 (at 192.168.202.58@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5195.234848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5195.247137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5195.267219] Lustre: Skipped 1 previous similar message [ 5195.278142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5197.796444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.797313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5197.815535] Lustre: Skipped 1 previous similar message [ 5200.172269] Lustre: server umount lustre-OST0000 complete [ 5204.370193] LustreError: 168826:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5204.390582] Lustre: 168172:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5204.396845] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5204.412179] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5204.424660] LustreError: 168172:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ccc23a54 x1877983226807040/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5204.441956] LustreError: 168172:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5204.450914] LustreError: 168172:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5204.461881] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5204.779824] Lustre: server umount lustre-MDT0000 complete [ 5208.035705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.049536] LustreError: 165085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.067805] LustreError: 165085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 5209.237297] LustreError: 165827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790984507 with bad export cookie 4753392659385584839 [ 5209.237718] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5209.251054] LustreError: 165827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5211.755469] Lustre: server umount lustre-MDT0001 complete [ 5225.016287] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5227.622854] Key type lgssc unregistered [ 5227.939426] LNet: 169577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5227.949630] LNetError: 169577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5227.962995] LNet: Removed LNI 192.168.202.158@tcp [ 5229.064172] Key type .llcrypt unregistered [ 5229.067287] Key type ._llcrypt unregistered [ 5254.686937] Key type ._llcrypt registered [ 5254.689187] Key type .llcrypt registered [ 5254.781416] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 5256.018334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5256.249381] alg: No test for adler32 (adler32-zlib) [ 5257.366342] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 5257.558953] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 5259.223298] Key type lgssc registered [ 5260.468257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5267.694884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5267.732916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5269.293321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5273.935602] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5279.713290] LustreError: 171005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5279.721449] LustreError: 171005:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5283.303800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5283.817136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5288.542578] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5295.683562] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5303.555650] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5312.032312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5312.561235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5314.621096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5320.050183] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5327.572599] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5335.010309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5335.020314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5335.041530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5336.726733] Lustre: server umount lustre-OST0000 complete [ 5344.230447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.232271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5344.243551] Lustre: Skipped 1 previous similar message [ 5344.258928] Lustre: Skipped 1 previous similar message [ 5346.928521] Lustre: server umount lustre-MDT0000 complete [ 5349.353060] LustreError: 171005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5349.364115] LustreError: 171005:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5351.055619] LustreError: 170985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790984648 with bad export cookie 16513445137719772009 [ 5351.074712] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5360.761429] Lustre: server umount lustre-MDT0001 complete [ 5371.260176] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5374.032496] Key type lgssc unregistered [ 5374.275989] LNet: 174536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5374.288460] LNetError: 174536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5374.299901] LNet: Removed LNI 192.168.202.158@tcp [ 5375.056953] Key type .llcrypt unregistered [ 5375.059932] Key type ._llcrypt unregistered [ 5410.877810] Key type ._llcrypt registered [ 5410.880593] Key type .llcrypt registered [ 5411.036607] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 5412.364092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5412.395182] alg: No test for adler32 (adler32-zlib) [ 5413.526490] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 5413.729471] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 5415.401326] Key type lgssc registered [ 5416.776267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5424.138239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5424.155835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.637984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5430.222444] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5435.875569] LustreError: 176256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5435.909672] LustreError: 176256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5438.873814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5439.384208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5444.331714] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5451.546148] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5456.358770] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5464.782275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5465.086631] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5465.208402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5470.366708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5472.437733] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5480.043328] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5491.170148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5491.173500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5491.183128] LustreError: Skipped 1 previous similar message [ 5491.185286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5491.204171] Lustre: Skipped 1 previous similar message [ 5491.230220] Lustre: Skipped 1 previous similar message [ 5495.651342] Lustre: server umount lustre-OST0000 complete [ 5500.899177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5500.908988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.922061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5506.384868] Lustre: server umount lustre-MDT0000 complete [ 5506.538967] LustreError: 176983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5506.556142] LustreError: 176983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5511.050137] LustreError: 176235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790984808 with bad export cookie 6667883179607764317 [ 5511.064703] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5511.084352] LustreError: 176235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5511.567509] Lustre: server umount lustre-MDT0001 complete [ 5524.736321] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5527.731959] Key type lgssc unregistered [ 5528.009109] LNet: 179788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5528.018427] LNetError: 179788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5528.031623] LNet: Removed LNI 192.168.202.158@tcp [ 5528.891381] Key type .llcrypt unregistered [ 5528.895715] Key type ._llcrypt unregistered [ 5553.547171] Key type ._llcrypt registered [ 5553.550480] Key type .llcrypt registered [ 5553.663587] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 5555.092334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5555.216809] alg: No test for adler32 (adler32-zlib) [ 5556.442252] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 5556.802299] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 5558.480805] Key type lgssc registered [ 5560.152203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5572.864356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5572.884787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5574.260323] Lustre: *** cfs_fail_loc=135, val=0*** [ 5574.349776] LustreError: 181331:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5574.355708] LustreError: 181331:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.158@tcp: cfg command failed: rc = -2 [ 5574.375080] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5574.375080] [ 5574.394894] LustreError: MGC192.168.202.158@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5574.411944] LustreError: 181292:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5574.427581] LustreError: 181292:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5574.433785] LustreError: 181292:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5574.442469] LustreError: 181292:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5574.512220] Lustre: server umount lustre-MDT0000 complete [ 5574.514271] LustreError: 181292:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5581.907428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.423044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5586.915528] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5592.544451] LustreError: 181703:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5592.557289] LustreError: 181703:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5597.209828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5597.470043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5602.264224] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5610.464278] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5616.158484] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5622.432830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.453864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5624.957821] Lustre: server umount lustre-MDT0000 complete [ 5627.881313] LustreError: 181702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5627.897447] LustreError: 181702:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5629.318151] LustreError: 181684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790984927 with bad export cookie 2348080227441044937 [ 5629.329348] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5629.336424] LustreError: 181684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5629.650727] Lustre: server umount lustre-MDT0001 complete [ 5642.249726] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5644.988343] Key type lgssc unregistered [ 5645.207844] LNet: 184025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5645.211566] LNetError: 184025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5646.262744] LNet: Removed LNI 192.168.202.158@tcp [ 5647.235154] Key type .llcrypt unregistered [ 5647.240046] Key type ._llcrypt unregistered [ 5670.649377] Key type ._llcrypt registered [ 5670.654355] Key type .llcrypt registered [ 5670.786899] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5671.051692] Key type .llcrypt unregistered [ 5671.054599] Key type ._llcrypt unregistered [ 5687.824377] Key type ._llcrypt registered [ 5687.830479] Key type .llcrypt registered [ 5688.044930] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 5689.581825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5689.622116] alg: No test for adler32 (adler32-zlib) [ 5690.753056] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 5691.083871] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 5692.807469] Key type lgssc registered [ 5694.258475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5702.749513] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5702.787114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5718.496756] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5723.615617] LustreError: 186169:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.158@tcp: failed processing log, type 4: rc = -110 [ 5750.303303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5757.312375] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5764.515701] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5771.660150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5772.169197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5772.234606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5776.237047] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5777.388779] LustreError: 186195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.417964] LustreError: 186195:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5782.497910] LustreError: 186194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5784.550661] LustreError: 187208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5784.590732] LustreError: 187208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5785.433645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5785.802540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5789.791386] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5799.038797] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5806.976746] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5810.297699] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5821.939039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5821.943215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5821.951044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5821.970776] Lustre: Skipped 1 previous similar message [ 5827.044918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5827.057972] Lustre: Skipped 1 previous similar message [ 5827.923486] Lustre: server umount lustre-OST0000 complete [ 5832.164217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5832.174765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5832.198649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5837.285815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5837.300668] Lustre: Skipped 2 previous similar messages [ 5837.794924] Lustre: server umount lustre-MDT0000 complete [ 5842.207822] LustreError: 188832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790985140 with bad export cookie 11853092646311556221 [ 5842.214426] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5842.225991] LustreError: 188832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5842.579304] Lustre: server umount lustre-MDT0001 complete [ 5853.648765] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 5857.066386] Key type lgssc unregistered [ 5857.364643] LNet: 190135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5857.374275] LNetError: 190135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5857.401792] LNet: Removed LNI 192.168.202.158@tcp [ 5858.314253] Key type .llcrypt unregistered [ 5858.316879] Key type ._llcrypt unregistered [ 5884.992974] Key type ._llcrypt registered [ 5884.996938] Key type .llcrypt registered [ 5885.142159] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 5885.952397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5886.030524] alg: No test for adler32 (adler32-zlib) [ 5887.515878] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 5887.998883] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 5889.735352] Key type lgssc registered [ 5891.253265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5898.435679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5898.458950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5900.009743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5904.590496] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5909.480764] LustreError: 191564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5909.527260] LustreError: 191564:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5913.889836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5914.331040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5918.422554] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5925.646212] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5931.089532] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5937.970415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5938.329988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5944.527975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5944.874855] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5951.724482] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5958.385819] Lustre: Failing over lustre-MDT0000 [ 5958.834838] Lustre: server umount lustre-MDT0000 complete [ 5959.808676] LustreError: 191559:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5960.181378] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.205573] Lustre: Skipped 1 previous similar message [ 5964.885468] LustreError: 191558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5964.898114] LustreError: 191558:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5970.013238] LustreError: 191558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.038936] LustreError: 191558:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5975.132481] LustreError: 191558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5975.157107] LustreError: 191558:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5976.543281] Lustre: 190865:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790985258/real 1790985258] req@0000000080d7d432 x1877984081371904/t0(0) o400->MGC192.168.202.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790985274 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5976.560924] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5977.486434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5986.405502] LustreError: 191559:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5986.445255] LustreError: 191559:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 5986.791159] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd029952c0a8e25cf [ 5986.820235] Lustre: MGC192.168.202.158@tcp: Connection restored to 0@lo (at 0@lo) [ 5987.086089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5987.147272] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5991.531795] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5992.257743] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 5992.435930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5992.466842] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5992.541982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5996.744179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6006.243991] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6021.601312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6021.607901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6021.616910] Lustre: Skipped 1 previous similar message [ 6021.622654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6023.141539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6023.162622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.262150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6028.275337] Lustre: Skipped 2 previous similar messages [ 6033.375173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6033.387248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6033.401886] Lustre: Skipped 1 previous similar message [ 6033.654893] Lustre: server umount lustre-OST0000 complete [ 6038.500657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6038.503764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6038.515773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6038.531897] Lustre: Skipped 1 previous similar message [ 6038.551121] Lustre: Skipped 1 previous similar message [ 6048.739237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6048.765413] Lustre: Skipped 3 previous similar messages [ 6052.319135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6052.559818] Lustre: server umount lustre-MDT0000 complete [ 6053.857359] LustreError: 192292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6053.867982] LustreError: 192292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6056.136020] LustreError: 195441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790985354 with bad export cookie 14999684050322335183 [ 6056.161425] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6056.170742] LustreError: 195441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6056.777303] Lustre: server umount lustre-MDT0001 complete [ 6068.343985] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 6071.012456] Key type lgssc unregistered [ 6071.190674] LNet: 196521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6071.195487] LNetError: 196521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6071.206763] LNet: Removed LNI 192.168.202.158@tcp [ 6071.914227] Key type .llcrypt unregistered [ 6071.916657] Key type ._llcrypt unregistered [ 6092.986575] Key type ._llcrypt registered [ 6092.988770] Key type .llcrypt registered [ 6093.072206] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 6094.137245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6094.158947] alg: No test for adler32 (adler32-zlib) [ 6095.180953] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 6095.513537] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 6097.263260] Key type lgssc registered [ 6098.707356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6106.184192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6106.217615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.680323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6112.545447] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6117.866267] LustreError: 197928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6117.889061] LustreError: 197928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6122.975953] LustreError: 197929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6123.022323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6123.546270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6128.992643] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6136.389633] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6143.147454] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6150.543468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6150.870561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6152.974504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6157.867761] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6164.898043] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6167.542073] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6174.200293] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6193.490222] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6199.264213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6199.273679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.277027] LustreError: Skipped 1 previous similar message [ 6199.284159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6199.290347] Lustre: Skipped 1 previous similar message [ 6199.305535] Lustre: Skipped 1 previous similar message [ 6204.384513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6204.398393] Lustre: Skipped 1 previous similar message [ 6205.472330] Lustre: server umount lustre-OST0000 complete [ 6209.504691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6209.510944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6209.524421] Lustre: Skipped 1 previous similar message [ 6209.544518] Lustre: Skipped 1 previous similar message [ 6214.625432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6214.630359] Lustre: Skipped 1 previous similar message [ 6219.747177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6219.758255] Lustre: Skipped 1 previous similar message [ 6223.328247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6223.554551] Lustre: server umount lustre-MDT0000 complete [ 6224.886975] LustreError: 197924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6224.900265] LustreError: 197924:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6228.395947] LustreError: 198663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790985526 with bad export cookie 6655972550143813664 [ 6228.397614] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6228.409104] LustreError: 198663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6228.912687] Lustre: server umount lustre-MDT0001 complete [ 6243.038694] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 6245.510825] Key type lgssc unregistered [ 6245.876586] LNet: 201797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6245.885548] LNetError: 201797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6245.903902] LNet: Removed LNI 192.168.202.158@tcp [ 6246.919886] Key type .llcrypt unregistered [ 6246.923276] Key type ._llcrypt unregistered [ 6266.615291] Key type ._llcrypt registered [ 6266.618915] Key type .llcrypt registered [ 6266.758618] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 6267.802959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6268.013178] alg: No test for adler32 (adler32-zlib) [ 6269.036860] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 6269.234424] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 6270.903284] Key type lgssc registered [ 6272.005281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6280.767824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6280.797145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6282.364990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6286.966492] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6292.965574] LustreError: 203078:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.983916] LustreError: 203078:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6295.110270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6295.468454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6300.079980] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6307.247879] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6314.734233] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6322.272469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6322.712049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6327.790387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6330.033958] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6340.303318] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6347.659376] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6357.432521] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6364.518652] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6374.105961] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6387.169634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.171827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6387.189032] Lustre: Skipped 1 previous similar message [ 6387.218962] Lustre: Skipped 1 previous similar message [ 6391.712407] Lustre: server umount lustre-OST0000 complete [ 6397.408138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6397.411268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6397.443764] Lustre: Skipped 1 previous similar message [ 6397.480472] Lustre: Skipped 1 previous similar message [ 6402.528865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6402.948808] Lustre: server umount lustre-MDT0000 complete [ 6407.497461] LustreError: 205756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790985705 with bad export cookie 5725507005412597141 [ 6407.499651] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6407.517226] LustreError: 205756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6407.962290] Lustre: server umount lustre-MDT0001 complete [ 6420.992303] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 6424.441478] Key type lgssc unregistered [ 6424.675217] LNet: 208015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6424.681798] LNetError: 208015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6425.713726] LNet: Removed LNI 192.168.202.158@tcp [ 6426.871657] Key type .llcrypt unregistered [ 6426.876154] Key type ._llcrypt unregistered [ 6457.328949] Key type ._llcrypt registered [ 6457.331688] Key type .llcrypt registered [ 6457.452418] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 6458.712085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6458.771524] alg: No test for adler32 (adler32-zlib) [ 6460.040709] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 6460.329722] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 6462.055671] Key type lgssc registered [ 6463.124060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6471.222678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6471.238220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6472.822263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6478.856735] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6482.412762] LustreError: 209446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.442907] LustreError: 209446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6487.527763] LustreError: 209447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6488.779235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.085828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6494.292536] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6501.737565] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6507.714958] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6514.047560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6514.537209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6517.630752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6521.507734] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6528.386988] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6543.329455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6543.332657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.342508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6544.353376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6544.357859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6544.364392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6544.377676] Lustre: Skipped 1 previous similar message [ 6548.749684] Lustre: server umount lustre-OST0000 complete [ 6554.601635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6554.607313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6554.631373] Lustre: Skipped 1 previous similar message [ 6554.655326] Lustre: Skipped 1 previous similar message [ 6559.024370] Lustre: server umount lustre-MDT0000 complete [ 6559.716544] LustreError: 209447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6559.734380] LustreError: 209447:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6562.445906] LustreError: 209427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790985860 with bad export cookie 16657949795723229802 [ 6562.450089] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6562.457475] LustreError: 209427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6562.939399] Lustre: server umount lustre-MDT0001 complete [ 6574.989173] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 6577.549731] Key type lgssc unregistered [ 6577.836438] LNet: 213130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6577.845276] LNetError: 213130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6577.863930] LNet: Removed LNI 192.168.202.158@tcp [ 6578.704321] Key type .llcrypt unregistered [ 6578.705635] Key type ._llcrypt unregistered [ 6596.348872] Key type ._llcrypt registered [ 6596.351294] Key type .llcrypt registered [ 6596.488911] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 6597.882466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6597.912938] alg: No test for adler32 (adler32-zlib) [ 6599.283358] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 6599.590862] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 6601.255219] Key type lgssc registered [ 6602.571537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6610.295406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6610.325144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.711752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6616.410510] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6621.152419] LustreError: 214409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6621.174649] LustreError: 214409:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6625.764216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6626.408035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6630.934798] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6638.555672] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6644.402068] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6650.965281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6651.559382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6656.782791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6657.752330] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6665.002787] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6682.593567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6682.597894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6682.604708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6682.642708] Lustre: Skipped 1 previous similar message [ 6684.893925] Lustre: server umount lustre-OST0000 complete [ 6692.832773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6692.837690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6692.846936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6692.874798] Lustre: Skipped 1 previous similar message [ 6692.886611] Lustre: Skipped 2 previous similar messages [ 6695.552894] Lustre: server umount lustre-MDT0000 complete [ 6697.956699] LustreError: 214409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6697.968435] LustreError: 214409:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6698.882907] LustreError: 214389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790985996 with bad export cookie 11555020345855555225 [ 6698.896676] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6699.321362] Lustre: server umount lustre-MDT0001 complete [ 6710.218858] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 6713.287311] Key type lgssc unregistered [ 6713.629622] LNet: 218091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6713.635031] LNetError: 218091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6713.664502] LNet: Removed LNI 192.168.202.158@tcp [ 6714.647652] Key type .llcrypt unregistered [ 6714.650173] Key type ._llcrypt unregistered [ 6734.203439] Key type ._llcrypt registered [ 6734.205257] Key type .llcrypt registered [ 6734.299309] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 6735.671772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6735.693166] alg: No test for adler32 (adler32-zlib) [ 6736.642424] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 6736.876587] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 6738.551619] Key type lgssc registered [ 6740.045626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6747.385905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6747.433053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6748.856380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6752.702465] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6758.371890] LustreError: 219371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6758.392832] LustreError: 219371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6760.876859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6761.356573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6765.844425] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6772.811242] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6777.845735] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6784.648516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6785.130162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6790.926927] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6793.329880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6797.903913] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6807.490358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6807.562786] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6807.654951] Lustre: lustre-OST0001: Not available for connect from 192.168.202.58@tcp (not set up) [ 6807.841205] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6814.548205] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6823.025361] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6824.724965] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6836.346188] Lustre: Permanently deactivating lustre-OST0001 [ 6836.363553] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6846.553704] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6852.351993] Lustre: Permanently reactivating lustre-OST0001 [ 6852.369337] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6852.380854] Lustre: Skipped 2 previous similar messages [ 6852.807396] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.820615] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6852.825690] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6852.834449] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6852.858535] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6863.332252] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6863.336141] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.343652] LustreError: Skipped 1 previous similar message [ 6863.345553] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6863.357087] Lustre: Skipped 2 previous similar messages [ 6865.381275] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6865.394123] Lustre: Skipped 2 previous similar messages [ 6868.754135] Lustre: server umount lustre-OST0001 complete [ 6870.499179] LustreError: 221672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6870.510455] LustreError: 221672:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6873.568994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6873.582920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6873.613946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6875.624959] LustreError: 221193:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6875.627842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6875.628710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6875.656174] LustreError: 221193:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6879.605472] Lustre: server umount lustre-OST0000 complete [ 6884.320037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6884.326658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6884.367904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6884.382361] Lustre: Skipped 2 previous similar messages [ 6889.884252] Lustre: server umount lustre-MDT0000 complete [ 6890.981507] LustreError: 219372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6890.999888] LustreError: 219372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6893.919784] LustreError: 219352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790986191 with bad export cookie 6509106650701030577 [ 6893.923592] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6893.935748] LustreError: 219352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6894.426959] Lustre: server umount lustre-MDT0001 complete [ 6914.305581] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 6917.723942] Key type lgssc unregistered [ 6918.198737] LNet: 224459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6918.205533] LNetError: 224459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6918.231859] LNet: Removed LNI 192.168.202.158@tcp [ 6919.413854] Key type .llcrypt unregistered [ 6919.419404] Key type ._llcrypt unregistered [ 6968.296742] Key type ._llcrypt registered [ 6968.299367] Key type .llcrypt registered [ 6968.404382] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 6969.880635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6970.015663] alg: No test for adler32 (adler32-zlib) [ 6971.223445] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 6971.470784] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 6973.207225] Key type lgssc registered [ 6974.643601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6982.877591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6982.903739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.276430] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6984.303669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6984.500760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6988.660310] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 6998.028892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6998.206336] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6998.265783] Lustre: 226680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6998.724411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7004.398758] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7014.367785] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7021.973608] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7030.989497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7031.163308] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7031.360696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7038.686897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7038.931654] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7047.005617] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7049.686929] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7056.357548] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7063.768614] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7069.929332] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7085.732394] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7085.747290] Lustre: Skipped 1 previous similar message [ 7105.576104] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7105.581881] Lustre: Skipped 2 previous similar messages [ 7138.406303] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7138.411209] Lustre: Skipped 4 previous similar messages [ 7207.660353] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7207.671845] Lustre: Skipped 7 previous similar messages [ 7218.173900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7218.175460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7218.184881] Lustre: Skipped 1 previous similar message [ 7218.186123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7218.213525] LustreError: Skipped 1 previous similar message [ 7222.712670] Lustre: server umount lustre-OST0000 complete [ 7228.388802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7228.406103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7228.409216] Lustre: Skipped 1 previous similar message [ 7228.440040] Lustre: Skipped 2 previous similar messages [ 7233.505749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7238.627508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7238.630361] Lustre: Skipped 2 previous similar messages [ 7242.207368] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7242.667107] Lustre: server umount lustre-MDT0000 complete [ 7248.453454] LustreError: 229248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790986546 with bad export cookie 9797481496099345190 [ 7248.475647] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7248.476817] LustreError: 229248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7249.532659] Lustre: server umount lustre-MDT0001 complete [ 7265.577714] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 7269.455766] Key type lgssc unregistered [ 7269.686705] LNet: 231506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7269.691800] LNetError: 231506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7270.761641] LNet: Removed LNI 192.168.202.158@tcp [ 7271.903988] Key type .llcrypt unregistered [ 7271.911399] Key type ._llcrypt unregistered [ 7300.837868] Key type ._llcrypt registered [ 7300.845764] Key type .llcrypt registered [ 7301.040206] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 7302.493505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7302.656427] alg: No test for adler32 (adler32-zlib) [ 7303.955746] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 7304.328939] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 7305.983245] Key type lgssc registered [ 7308.007281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7318.506675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7318.538902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7320.142359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7325.926339] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7329.270356] LustreError: 232935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.301414] LustreError: 232935:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7334.368194] LustreError: 232936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7336.765547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7337.259820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7342.127797] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7350.195157] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7357.257438] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7364.851100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7365.097510] LustreError: 234755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7365.497170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7370.764420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7372.300203] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_default_debug -1 all [ 7380.349113] Lustre: DEBUG MARKER: oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7396.330551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7396.342941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7396.359195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7398.372069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7398.373379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7398.399818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7402.088086] Lustre: server umount lustre-OST0000 complete [ 7408.609613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7408.615322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7408.642140] Lustre: Skipped 1 previous similar message [ 7408.665886] Lustre: Skipped 2 previous similar messages [ 7413.721798] Lustre: server umount lustre-MDT0000 complete [ 7413.737617] LustreError: 232935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7413.762579] LustreError: 232935:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7417.688025] LustreError: 235516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790986715 with bad export cookie 12174939948726319495 [ 7417.703676] LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7417.704063] LustreError: 235516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7417.963967] Lustre: server umount lustre-MDT0001 complete [ 7428.159343] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 7430.950356] Key type lgssc unregistered [ 7431.251340] LNet: 236624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7431.260340] LNetError: 236624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7431.287323] LNet: Removed LNI 192.168.202.158@tcp [ 7432.292419] Key type .llcrypt unregistered [ 7432.295881] Key type ._llcrypt unregistered [ 7455.176682] Key type ._llcrypt registered [ 7455.178875] Key type .llcrypt registered [ 7455.292065] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 7456.307084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7456.463943] alg: No test for adler32 (adler32-zlib) [ 7457.560409] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 7457.788810] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 7459.471664] Key type lgssc registered [ 7460.620755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7732.645735] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 7735.781693] Key type lgssc unregistered [ 7736.164812] LNet: 238666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7736.169961] LNetError: 238666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7736.194214] LNet: Removed LNI 192.168.202.158@tcp [ 7737.014656] Key type .llcrypt unregistered [ 7737.018815] Key type ._llcrypt unregistered [ 7763.044527] Key type ._llcrypt registered [ 7763.048712] Key type .llcrypt registered [ 7763.172090] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing set_hostid [ 7779.544521] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing load_modules_local [ 7781.021158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7781.074480] alg: No test for adler32 (adler32-zlib) [ 7782.225915] Lustre: Lustre: Build Version: 2.17.59_2_ge06ed12 [ 7782.685340] LNet: Added LNI 192.168.202.158@tcp [8/256/0/180] [ 7784.384402] Key type lgssc registered [ 7786.360307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7817.663597] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7484 sec ========== 20:25:13 (1790987113) [ 7819.495496] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:25:15 (1790987115) === [ 7823.480757] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:25:19 (1790987119) === [ 7862.449428] Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local [ 7865.080393] Key type lgssc unregistered [ 7865.348595] LNet: 243919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7865.352255] LNetError: 243919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7865.363645] LNet: Removed LNI 192.168.202.158@tcp [ 7866.072700] Key type .llcrypt unregistered [ 7866.074552] Key type ._llcrypt unregistered