[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 528968096 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003098] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.012096] LSM: Security Framework initializing [ 0.014046] Yama: becoming mindful. [ 0.015035] SELinux: Initializing. [ 0.016072] *** VALIDATE selinux *** [ 0.024187] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029359] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034047] *** VALIDATE tmpfs *** [ 0.035340] *** VALIDATE proc *** [ 0.037134] *** VALIDATE cgroup *** [ 0.038011] *** VALIDATE cgroup2 *** [ 0.040217] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044028] Spectre V2 : User space: Vulnerable [ 0.045009] Speculative Store Bypass: Vulnerable [ 0.048233] debug: unmapping init [mem 0xffffffffaa459000-0xffffffffaa460fff] [ 0.050187] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051719] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052024] ... version: 2 [ 0.053012] ... bit width: 48 [ 0.054010] ... generic registers: 4 [ 0.055011] ... value mask: 0000ffffffffffff [ 0.056018] ... max period: 00007fffffffffff [ 0.057016] ... fixed-purpose events: 3 [ 0.058014] ... event mask: 000000070000000f [ 0.059289] rcu: Hierarchical SRCU implementation. [ 0.061456] smp: Bringing up secondary CPUs ... [ 0.062623] x86: Booting SMP configuration: [ 0.063026] .... node #0, CPUs: #1 #2 #3 [ 0.067511] smp: Brought up 1 node, 4 CPUs [ 0.069015] smpboot: Max logical packages: 1 [ 0.070019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.153186] node 0 deferred pages initialised in 81ms [ 0.156118] devtmpfs: initialized [ 0.158237] x86/mm: Memory block size: 128MB [ 0.162635] gcov: version magic: 0x41383552 [ 0.164204] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165069] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166274] pinctrl core: initialized pinctrl subsystem [ 0.167191] [ 0.167628] ************************************************************* [ 0.168015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169017] ** ** [ 0.170012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171013] ** ** [ 0.172023] ** This means that this kernel is built to expose internal ** [ 0.173011] ** IOMMU data structures, which may compromise security on ** [ 0.174012] ** your system. ** [ 0.175012] ** ** [ 0.176012] ** If you see this message and you are not debugging the ** [ 0.177012] ** kernel, report this immediately to your vendor! ** [ 0.178010] ** ** [ 0.179014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180015] ************************************************************* [ 0.181704] NET: Registered protocol family 16 [ 0.182440] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186048] cpuidle: using governor menu [ 0.187876] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.191537] PCI: Using configuration type 1 for base access [ 0.193207] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.205254] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.213153] cryptd: max_cpu_qlen set to 1000 [ 0.216258] ACPI: Added _OSI(Module Device) [ 0.217014] ACPI: Added _OSI(Processor Device) [ 0.218011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223302] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226569] ACPI: Interpreter enabled [ 0.227059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228011] ACPI: Using IOAPIC for interrupt routing [ 0.229105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.230301] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239583] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.240026] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.241021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.242075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.244137] acpiphp: Slot [2] registered [ 0.245104] acpiphp: Slot [5] registered [ 0.246104] acpiphp: Slot [6] registered [ 0.247131] acpiphp: Slot [7] registered [ 0.248135] acpiphp: Slot [8] registered [ 0.249119] acpiphp: Slot [9] registered [ 0.250094] acpiphp: Slot [10] registered [ 0.251119] acpiphp: Slot [3] registered [ 0.252092] acpiphp: Slot [4] registered [ 0.253096] acpiphp: Slot [11] registered [ 0.254146] acpiphp: Slot [12] registered [ 0.255145] acpiphp: Slot [13] registered [ 0.256116] acpiphp: Slot [14] registered [ 0.257075] acpiphp: Slot [15] registered [ 0.258105] acpiphp: Slot [16] registered [ 0.259095] acpiphp: Slot [17] registered [ 0.260132] acpiphp: Slot [18] registered [ 0.261073] acpiphp: Slot [19] registered [ 0.262064] acpiphp: Slot [20] registered [ 0.265150] acpiphp: Slot [21] registered [ 0.266087] acpiphp: Slot [22] registered [ 0.267190] acpiphp: Slot [23] registered [ 0.268107] acpiphp: Slot [24] registered [ 0.269109] acpiphp: Slot [25] registered [ 0.270114] acpiphp: Slot [26] registered [ 0.271091] acpiphp: Slot [27] registered [ 0.272096] acpiphp: Slot [28] registered [ 0.273100] acpiphp: Slot [29] registered [ 0.274132] acpiphp: Slot [30] registered [ 0.275144] acpiphp: Slot [31] registered [ 0.277109] PCI host bridge to bus 0000:00 [ 0.279182] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.283032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.287056] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.295023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.299032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.301463] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.306535] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.310815] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.323694] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.328872] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.331022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.333025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.335021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.339217] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.342203] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.347068] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.352889] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.359023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.374022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.380020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.390131] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.403016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.409017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.427019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.437624] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.445022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.456022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.476022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.488045] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.495017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.504018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.522018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.531993] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.546026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.554035] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.582022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.594221] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.601019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.613020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.647021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.663824] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.674025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.685029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.706025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.723203] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.726380] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.728400] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.730419] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.733215] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.739172] iommu: Default domain type: Passthrough [ 0.740548] SCSI subsystem initialized [ 0.743121] ACPI: bus type USB registered [ 0.744125] usbcore: registered new interface driver usbfs [ 0.746073] usbcore: registered new interface driver hub [ 0.748084] usbcore: registered new device driver usb [ 0.751239] pps_core: LinuxPPS API ver. 1 registered [ 0.753012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.756063] PTP clock support registered [ 0.759098] EDAC MC: Ver: 3.0.0 [ 0.762175] PCI: Using ACPI for IRQ routing [ 0.765472] NetLabel: Initializing [ 0.767017] NetLabel: domain hash size = 128 [ 0.769016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.772171] NetLabel: unlabeled traffic allowed by default [ 0.775184] vgaarb: loaded [ 0.777344] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.779024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.787000] clocksource: Switched to clocksource kvm-clock [ 0.907902] VFS: Disk quotas dquot_6.6.0 [ 0.909747] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.914578] *** VALIDATE ramfs *** [ 0.916252] *** VALIDATE hugetlbfs *** [ 0.918259] pnp: PnP ACPI init [ 0.921225] pnp: PnP ACPI: found 6 devices [ 0.939195] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.943387] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.945715] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.948225] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.950982] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.954230] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.957849] NET: Registered protocol family 2 [ 0.960874] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.966979] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.971774] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.977885] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.981805] TCP: Hash tables configured (established 65536 bind 65536) [ 0.985119] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.989058] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.991338] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.993928] NET: Registered protocol family 1 [ 0.996474] RPC: Registered named UNIX socket transport module. [ 1.000441] RPC: Registered udp transport module. [ 1.003457] RPC: Registered tcp transport module. [ 1.006429] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.009511] NET: Registered protocol family 44 [ 1.012245] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.015067] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.018408] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.021850] PCI: CLS 0 bytes, default 64 [ 1.025075] Unpacking initramfs... [ 2.466506] debug: unmapping init [mem 0xffff9a363cc54000-0xffff9a363ffbffff] [ 2.470335] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.473270] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.476976] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.040477] Initialise system trusted keyrings [ 3.042226] Key type blacklist registered [ 3.046335] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.056563] zbud: loaded [ 3.061332] *** VALIDATE nfs *** [ 3.062934] *** VALIDATE nfs4 *** [ 3.064819] pstore: using deflate compression [ 3.068683] Platform Keyring initialized [ 3.185080] NET: Registered protocol family 38 [ 3.187422] Key type asymmetric registered [ 3.189846] Asymmetric key parser 'x509' registered [ 3.192277] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.196231] io scheduler mq-deadline registered [ 3.198518] io scheduler kyber registered [ 3.200609] io scheduler bfq registered [ 3.202647] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.206327] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.210449] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.213776] ACPI: Power Button [PWRF] [ 3.222247] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.230296] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.250949] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.264195] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.298228] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.326594] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.354701] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.359624] Non-volatile memory driver v1.3 [ 3.361190] Linux agpgart interface v0.103 [ 3.403149] virtio_blk virtio1: [vda] 147088 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.408562] vda: detected capacity change from 0 to 75309056 [ 3.425697] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.429093] vdb: detected capacity change from 0 to 1073741824 [ 3.446714] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.450995] vdc: detected capacity change from 0 to 2621440000 [ 3.470519] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.473711] vdd: detected capacity change from 0 to 2621440000 [ 3.506222] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.508931] vde: detected capacity change from 0 to 4294967296 [ 3.525722] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.528308] vdf: detected capacity change from 0 to 4294967296 [ 3.537433] libphy: Fixed MDIO Bus: probed [ 3.543713] usbcore: registered new interface driver usbserial_generic [ 3.546039] usbserial: USB Serial support registered for generic [ 3.548748] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.552560] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.554201] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.556708] mousedev: PS/2 mouse device common for all mice [ 3.559638] rtc_cmos 00:05: RTC can wake from S4 [ 3.562653] rtc_cmos 00:05: registered as rtc0 [ 3.563908] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.566830] intel_pstate: CPU model not supported [ 3.569236] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.575539] hid: raw HID events driver (C) Jiri Kosina [ 3.578049] usbcore: registered new interface driver usbhid [ 3.580613] usbhid: USB HID core driver [ 3.582166] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.583038] drop_monitor: Initializing network drop monitor service [ 3.589175] Initializing XFRM netlink socket [ 3.591132] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.591207] NET: Registered protocol family 10 [ 3.598594] Segment Routing with IPv6 [ 3.600480] NET: Registered protocol family 17 [ 3.603329] mpls_gso: MPLS GSO support [ 3.612186] RAS: Correctable Errors collector initialized. [ 3.614356] AVX version of gcm_enc/dec engaged. [ 3.615868] AES CTR mode by8 optimization enabled [ 3.703524] sched_clock: Marking stable (3703491470, 0)->(4629327201, -925835731) [ 3.708150] registered taskstats version 1 [ 3.710366] Loading compiled-in X.509 certificates [ 3.712302] zswap: loaded using pool lzo/zbud [ 3.742364] Key type big_key registered [ 3.755175] Key type encrypted registered [ 3.757066] ima: No TPM chip found, activating TPM-bypass! [ 3.759093] ima: Allocated hash algorithm: sha1 [ 3.760926] ima: No architecture policies found [ 3.762758] evm: Initialising EVM extended attributes: [ 3.764816] evm: security.selinux [ 3.766036] evm: security.ima [ 3.767131] evm: security.capability [ 3.768725] evm: HMAC attrs: 0x1 [ 3.771125] rtc_cmos 00:05: setting system clock to 2026-08-27 19:59:50 UTC (1787860790) [ 3.777740] debug: unmapping init [mem 0xffffffffab403000-0xffffffffab5fffff] [ 3.781654] debug: unmapping init [mem 0xffffffffaa182000-0xffffffffaa458fff] [ 3.790547] Write protecting the kernel read-only data: 28672k [ 3.794024] debug: unmapping init [mem 0xffffffffa8803000-0xffffffffa89fffff] [ 3.797655] debug: unmapping init [mem 0xffffffffa9114000-0xffffffffa91fffff] [ 3.843095] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.851993] systemd[1]: Detected virtualization kvm. [ 3.854268] systemd[1]: Detected architecture x86-64. [ 3.856189] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.886715] systemd[1]: No hostname configured. [ 3.888815] systemd[1]: Set hostname to . [ 3.891125] random: systemd: uninitialized urandom read (16 bytes read) [ 3.893883] systemd[1]: Initializing machine ID from random generator. [ 4.040832] random: systemd: uninitialized urandom read (16 bytes read) [ 4.044125] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.049527] random: systemd: uninitialized urandom read (16 bytes read) [ 4.052262] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.058260] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.749469] device-mapper: uevent: version 1.0.3 [ 4.752422] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices.[ 5.488086] random: fast init done Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.569961] virtio_net virtio0 ens2: renamed from eth0 [ 5.660225] scsi host0: ata_piix [ 5.677280] scsi host1: ata_piix [ 5.679440] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.683191] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.287637] random: crng init done [ 10.301109] random: 7 urandom warning(s) missed due to ratelimiting [ 11.098219] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.040939] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.439704] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.568425] SELinux: Disabled at runtime. [ 17.702664] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.720523] systemd[1]: Detected virtualization kvm. [ 17.726094] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.430587] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.445596] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.457173] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.476131] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.489750] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.520863] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.554772] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 20.001990] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 21.045795] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.885592] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.939799] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.225060] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.297727] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 26.914981] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 27.816397] NFS: Registering the id_resolver key type [ 27.818780] Key type id_resolver registered [ 27.825398] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg444-server login: [ 58.753860] hrtimer: interrupt took 4784880 ns [ 99.355956] libcfs: loading out-of-tree module taints kernel. [ 99.382442] Key type ._llcrypt registered [ 99.384334] Key type .llcrypt registered [ 99.466664] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 118.859534] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 120.530217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 120.552859] alg: No test for adler32 (adler32-zlib) [ 122.260370] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 123.750775] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 125.599302] Key type lgssc registered [ 127.914282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.758724] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 194.544303] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 206.989637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 207.024032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.319607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 208.350338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 208.482259] Lustre: lustre-MDT0000: new disk, initializing [ 208.633670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.651238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 213.024945] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.115845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.213966] Lustre: 6545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 226.249292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 226.254872] Lustre: Skipped 1 previous similar message [ 226.328626] Lustre: lustre-MDT0001: new disk, initializing [ 226.390666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.413250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 226.427886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 231.402783] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.210662] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 245.623560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 246.080047] Lustre: lustre-OST0000: new disk, initializing [ 246.082696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 246.096711] Lustre: 8480:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 246.151788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 253.040504] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.019962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 255.026972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 255.130282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 267.459215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.623761] Lustre: lustre-OST0001: new disk, initializing [ 267.631575] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 267.641779] Lustre: 9553:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 267.747359] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 273.485956] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.004877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 275.009214] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 275.063401] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 286.354933] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 298.230737] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 304.361139] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing check_logdir /tmp/testlogs/ [ 309.474674] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing yml_node [ 313.569638] Lustre: DEBUG MARKER: Client: 2.17.57.120 [ 315.884900] Lustre: DEBUG MARKER: MDS: 2.17.57.120 [ 318.425731] Lustre: DEBUG MARKER: OSS: 2.17.57.120 [ 320.264213] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 16:05:05 EDT 2026 [ 339.081678] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 340.661302] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 359.392738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 359.404847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.429543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.953190] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.956917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.987214] Lustre: Skipped 3 previous similar messages [ 363.824524] Lustre: server umount lustre-MDT0000 complete [ 367.238346] LustreError: 6535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861153 with bad export cookie 3933477506886856657 [ 367.242691] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 367.248523] LustreError: 6535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 372.197474] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.197674] LustreError: 6552:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.205375] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.206085] Lustre: Skipped 3 previous similar messages [ 372.223925] LustreError: 6552:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 373.640868] Lustre: server umount lustre-MDT0001 complete [ 377.049706] Lustre: server umount lustre-OST0000 complete [ 380.686830] Lustre: server umount lustre-OST0001 complete [ 386.485844] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 396.260755] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 431.643551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.892385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 431.919209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 431.998115] Lustre: lustre-MDT0000: new disk, initializing [ 432.077930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.096361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 436.633905] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 447.928509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.042390] Lustre: 15107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 448.113391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 448.118558] Lustre: Skipped 1 previous similar message [ 448.212096] Lustre: lustre-MDT0001: new disk, initializing [ 448.450524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 448.518665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 448.535165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 453.674712] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 462.646514] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 468.431954] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 474.543953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 474.970637] Lustre: lustre-OST0000: new disk, initializing [ 474.976772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 474.981915] Lustre: 17074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 475.059494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.832954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 476.849229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 477.001644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 482.066251] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 491.763901] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 496.242831] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 496.498623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 501.063555] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 501.266522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 507.872589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 507.878187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.884568] LustreError: Skipped 1 previous similar message [ 507.900985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.901509] Lustre: Skipped 1 previous similar message [ 507.925695] Lustre: Skipped 2 previous similar messages [ 509.785152] Lustre: server umount lustre-OST0000 complete [ 515.042556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 515.061661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.090119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 527.343122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 527.353779] Lustre: Skipped 7 previous similar messages [ 528.353316] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 528.554404] Lustre: server umount lustre-MDT0000 complete [ 532.449872] LustreError: 16220:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.480731] LustreError: 16220:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 532.931066] LustreError: 15099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861319 with bad export cookie 3933477506886858190 [ 532.942981] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 532.945674] LustreError: 15099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 533.495112] Lustre: server umount lustre-MDT0001 complete [ 537.154874] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 16:08:42 (1787861322) [ 543.749994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.166874] LustreError: 19338:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.222737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.162554] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 549.351455] LustreError: 19339:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.448744] LustreError: 19338:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.854486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.253917] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 571.199778] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 578.395936] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 585.504327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 592.016632] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 599.342784] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 610.791537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 610.803359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.818811] Lustre: Skipped 1 previous similar message [ 610.832942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 616.456820] Lustre: server umount lustre-OST0000 complete [ 626.589819] Lustre: server umount lustre-MDT0000 complete [ 628.192239] LustreError: 19338:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.213954] LustreError: 19338:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 630.591945] LustreError: 19320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861417 with bad export cookie 3933477506886859072 [ 630.602501] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 631.185782] Lustre: server umount lustre-MDT0001 complete [ 643.712834] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 646.924172] Key type lgssc unregistered [ 647.241619] LNet: 22873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.250801] LNetError: 22873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.272863] LNet: Removed LNI 192.168.204.144@tcp [ 648.110144] Key type .llcrypt unregistered [ 648.111951] Key type ._llcrypt unregistered [ 678.240902] Key type ._llcrypt registered [ 678.242963] Key type .llcrypt registered [ 678.339483] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 679.604173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 679.822227] alg: No test for adler32 (adler32-zlib) [ 680.868679] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 681.215415] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 682.935180] Key type lgssc registered [ 684.373460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 692.709513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 692.730639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.247755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.110576] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 703.465736] LustreError: 24289:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.496196] LustreError: 24289:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 708.311765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 708.581979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 708.810391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 713.843133] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 721.875083] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 727.561601] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 733.687693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 734.006324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 739.134066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 739.655607] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 749.739827] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 770.018470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 770.033065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.053257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 771.439105] Lustre: server umount lustre-OST0000 complete [ 775.648852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 775.653895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.670386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.233955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.262647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.293315] Lustre: Skipped 1 previous similar message [ 781.361062] Lustre: server umount lustre-MDT0000 complete [ 784.353156] LustreError: 24284:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 784.363793] LustreError: 24284:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 785.195646] LustreError: 25023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861571 with bad export cookie 17585325799301322216 [ 785.198760] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.211174] LustreError: 25023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 794.781740] Lustre: server umount lustre-MDT0001 complete [ 806.112852] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 809.227957] Key type lgssc unregistered [ 809.591710] LNet: 28270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.597768] LNetError: 28270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.609769] LNet: Removed LNI 192.168.204.144@tcp [ 810.484545] Key type .llcrypt unregistered [ 810.487448] Key type ._llcrypt unregistered [ 833.281610] Key type ._llcrypt registered [ 833.284383] Key type .llcrypt registered [ 833.409884] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 834.643481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 834.681177] alg: No test for adler32 (adler32-zlib) [ 835.774772] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 836.015685] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 837.679900] Key type lgssc registered [ 839.131803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 848.615592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 848.627796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.941264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 855.195949] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 860.131481] LustreError: 29685:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.146869] LustreError: 29685:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 863.795312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 864.133022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 868.920596] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 878.185942] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 885.362394] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 898.766736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 899.167872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.628818] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 913.013150] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 919.660630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 925.163623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 925.170407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.171378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.192746] Lustre: Skipped 1 previous similar message [ 926.590260] Lustre: server umount lustre-OST0000 complete [ 930.277177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.282161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 930.295620] Lustre: Skipped 1 previous similar message [ 930.301969] Lustre: Skipped 1 previous similar message [ 935.394829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.407627] Lustre: Skipped 2 previous similar messages [ 936.544562] Lustre: server umount lustre-MDT0000 complete [ 940.413156] LustreError: 29668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861727 with bad export cookie 15876246871988083369 [ 940.426573] LustreError: 29668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 940.426750] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 940.799149] Lustre: server umount lustre-MDT0001 complete [ 952.354660] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 954.833815] Key type lgssc unregistered [ 955.089972] LNet: 33569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.106645] LNetError: 33569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.127334] LNet: Removed LNI 192.168.204.144@tcp [ 956.108070] Key type .llcrypt unregistered [ 956.112712] Key type ._llcrypt unregistered [ 984.226376] Key type ._llcrypt registered [ 984.230413] Key type .llcrypt registered [ 984.382335] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 985.686189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 985.763093] alg: No test for adler32 (adler32-zlib) [ 987.037576] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 987.450527] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 989.111171] Key type lgssc registered [ 990.650726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 999.420120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 999.441959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.094350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.139097] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1010.664459] LustreError: 34986:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1010.702295] LustreError: 34986:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1015.400480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.925776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1021.000454] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1029.311963] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1035.449959] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1043.155665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1043.654759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.754281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1050.098475] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1057.262126] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1070.058586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1070.068890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1070.089321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1071.610383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1071.619260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.622173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1071.661611] Lustre: Skipped 1 previous similar message [ 1076.706088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1076.725755] Lustre: Skipped 1 previous similar message [ 1081.861722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1081.869703] Lustre: Skipped 1 previous similar message [ 1082.336488] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1082.613272] Lustre: server umount lustre-OST0000 complete [ 1087.458347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1087.465795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.478456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1092.067065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.638314] Lustre: server umount lustre-MDT0000 complete [ 1096.675691] LustreError: 34967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861883 with bad export cookie 8619526343485033321 [ 1096.682957] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1096.685130] LustreError: 34967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1096.940761] Lustre: server umount lustre-MDT0001 complete [ 1107.977431] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1111.340718] Key type lgssc unregistered [ 1111.625065] LNet: 38525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.641356] LNetError: 38525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.662632] LNet: Removed LNI 192.168.204.144@tcp [ 1112.769993] Key type .llcrypt unregistered [ 1112.772179] Key type ._llcrypt unregistered [ 1135.803895] Key type ._llcrypt registered [ 1135.808698] Key type .llcrypt registered [ 1135.977914] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1137.154146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1137.246362] alg: No test for adler32 (adler32-zlib) [ 1138.313136] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 1138.618320] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1140.287157] Key type lgssc registered [ 1141.529690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1149.560843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1149.625987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.221980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.785618] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1160.676283] LustreError: 39931:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.700100] LustreError: 39931:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1163.487605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1163.736300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1166.606900] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1174.682322] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1184.198388] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1194.020366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1194.576799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.606246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1200.207565] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1206.432294] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1219.544854] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1220.577844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.584218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1220.599682] Lustre: Skipped 1 previous similar message [ 1220.618357] Lustre: Skipped 1 previous similar message [ 1224.634708] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1229.755149] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1229.765556] Lustre: Skipped 2 previous similar messages [ 1230.303168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1230.480836] Lustre: server umount lustre-OST0000 complete [ 1255.405345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.416463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1255.441888] Lustre: Skipped 1 previous similar message [ 1255.458475] Lustre: Skipped 1 previous similar message [ 1257.326717] Lustre: server umount lustre-MDT0000 complete [ 1260.519028] LustreError: 40659:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.551354] LustreError: 40659:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1262.518638] LustreError: 40666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862049 with bad export cookie 1251164059513606433 [ 1262.526587] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1262.535067] LustreError: 40666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1268.939084] Lustre: server umount lustre-MDT0001 complete [ 1280.249335] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1283.205460] Key type lgssc unregistered [ 1283.610304] LNet: 43644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.619439] LNetError: 43644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.641885] LNet: Removed LNI 192.168.204.144@tcp [ 1284.695778] Key type .llcrypt unregistered [ 1284.703113] Key type ._llcrypt unregistered [ 1314.665290] Key type ._llcrypt registered [ 1314.671444] Key type .llcrypt registered [ 1314.897860] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1315.953783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1315.994069] alg: No test for adler32 (adler32-zlib) [ 1317.336282] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 1317.807818] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1319.543868] Key type lgssc registered [ 1321.484859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1331.954539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1331.977308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.576244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.249655] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1343.969653] LustreError: 45063:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.998190] LustreError: 45063:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1348.072494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.683868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1353.816328] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1360.425862] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1366.547925] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1375.970753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1376.278553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.397580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1384.698211] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1393.820210] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1399.776978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1399.786971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1399.810224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1400.812193] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1404.386749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1404.389925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1404.409101] Lustre: Skipped 1 previous similar message [ 1404.440563] Lustre: Skipped 1 previous similar message [ 1406.254903] Lustre: server umount lustre-MDT0000 complete [ 1409.505549] LustreError: 46991:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.520485] LustreError: 46991:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1410.534535] LustreError: 45044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862197 with bad export cookie 12882878055396815969 [ 1410.543958] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1410.551686] LustreError: 45044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1410.998202] LustreError: 45059:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1411.030214] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 1411.045649] Lustre: Skipped 1 previous similar message [ 1414.625059] LustreError: 45060:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1414.626034] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.648814] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1415.659758] Lustre: Skipped 1 previous similar message [ 1417.025048] Lustre: server umount lustre-MDT0001 complete [ 1438.690124] Lustre: server umount lustre-OST0000 complete [ 1455.466762] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1458.340698] Key type lgssc unregistered [ 1458.790308] LNet: 48929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.803989] LNetError: 48929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.890043] LNet: Removed LNI 192.168.204.144@tcp [ 1460.696313] Key type .llcrypt unregistered [ 1460.701619] Key type ._llcrypt unregistered [ 1486.783907] Key type ._llcrypt registered [ 1486.786917] Key type .llcrypt registered [ 1486.893600] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1488.076187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1488.281375] alg: No test for adler32 (adler32-zlib) [ 1489.326366] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 1489.590987] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1491.305150] Key type lgssc registered [ 1492.769704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1501.523201] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1501.583321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1517.151604] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1522.272765] LustreError: 50316:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 1548.960612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.926688] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1566.198807] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1732.980977] Lustre: server umount lustre-OST0000 complete [ 1751.096254] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1754.507423] Key type lgssc unregistered [ 1754.924857] LNet: 51998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.931445] LNetError: 51998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.951055] LNet: Removed LNI 192.168.204.144@tcp [ 1755.936761] Key type .llcrypt unregistered [ 1755.941420] Key type ._llcrypt unregistered [ 1783.108802] Key type ._llcrypt registered [ 1783.110837] Key type .llcrypt registered [ 1783.220635] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1784.855793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1784.906248] alg: No test for adler32 (adler32-zlib) [ 1786.190397] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 1786.473334] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1788.175166] Key type lgssc registered [ 1789.665822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1797.562635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1797.592053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.147126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.537892] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1808.359403] LustreError: 53412:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1808.383852] LustreError: 53412:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1812.862339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.214554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1817.265634] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1824.931791] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1830.539511] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1837.335365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1837.776365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.887667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1844.815423] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1853.076474] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1859.575293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.583511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1859.595179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.608479] LustreError: Skipped 1 previous similar message [ 1859.612127] Lustre: Skipped 1 previous similar message [ 1864.675899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.680994] Lustre: Skipped 1 previous similar message [ 1869.792953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1869.808243] Lustre: Skipped 1 previous similar message [ 1870.303109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1870.507333] Lustre: server umount lustre-OST0000 complete [ 1880.032905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1880.036165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.055942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1880.075294] Lustre: Skipped 2 previous similar messages [ 1880.097809] Lustre: Skipped 1 previous similar message [ 1882.309210] Lustre: server umount lustre-MDT0000 complete [ 1885.152911] LustreError: 53412:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.171064] LustreError: 53412:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1885.664420] LustreError: 53392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862672 with bad export cookie 17581956889345333848 [ 1885.679935] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1885.950044] Lustre: server umount lustre-MDT0001 complete [ 1899.472885] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1902.649944] Key type lgssc unregistered [ 1902.959560] LNet: 56950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.969976] LNetError: 56950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.007375] LNet: Removed LNI 192.168.204.144@tcp [ 1903.934820] Key type .llcrypt unregistered [ 1903.936483] Key type ._llcrypt unregistered [ 1927.698243] Key type ._llcrypt registered [ 1927.699810] Key type .llcrypt registered [ 1927.889155] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1928.720548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1928.755611] alg: No test for adler32 (adler32-zlib) [ 1930.083737] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 1930.345653] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1931.975359] Key type lgssc registered [ 1933.209799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1944.576791] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1944.631208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1960.351972] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1965.471755] LustreError: 58341:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 1992.223212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2000.286771] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2007.506131] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2014.370596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.973758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.046475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2020.066646] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2020.332196] LustreError: 58366:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.374180] LustreError: 58366:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2025.441983] LustreError: 58367:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.472512] LustreError: 58366:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.499748] LustreError: 58366:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2029.184842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.575738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2033.951929] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2041.962652] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2048.173322] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2051.040590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2051.050930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2051.083270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2055.139618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.141986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2055.151221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2055.154437] Lustre: Skipped 1 previous similar message [ 2056.635732] Lustre: server umount lustre-OST0000 complete [ 2079.728521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2079.736958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2079.753777] Lustre: Skipped 1 previous similar message [ 2079.776141] Lustre: Skipped 1 previous similar message [ 2084.833488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.845219] Lustre: Skipped 1 previous similar message [ 2085.954505] Lustre: server umount lustre-MDT0000 complete [ 2089.953627] LustreError: 59381:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.965682] LustreError: 59381:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2090.802939] LustreError: 58348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862877 with bad export cookie 12690333353334833494 [ 2090.816966] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2090.821269] LustreError: 58348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2105.311696] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2105.577477] Lustre: server umount lustre-MDT0001 complete [ 2119.481614] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2122.698779] Key type lgssc unregistered [ 2123.194425] LNet: 62097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.213614] LNetError: 62097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.255822] LNet: Removed LNI 192.168.204.144@tcp [ 2124.017617] Key type .llcrypt unregistered [ 2124.022385] Key type ._llcrypt unregistered [ 2148.222780] Key type ._llcrypt registered [ 2148.225575] Key type .llcrypt registered [ 2148.331965] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2149.470545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2149.524222] alg: No test for adler32 (adler32-zlib) [ 2150.633517] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 2150.939205] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2152.610197] Key type lgssc registered [ 2153.853130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2162.083943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2162.143630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.796939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.107867] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2173.928752] LustreError: 63507:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2173.947977] LustreError: 63507:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2178.229971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.770765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2183.732346] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2191.819247] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2198.189830] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2204.992140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2205.362862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.467715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2211.347853] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2218.860896] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2240.486301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2240.487053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.510809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2240.522307] Lustre: Skipped 1 previous similar message [ 2240.539117] Lustre: Skipped 1 previous similar message [ 2242.286922] Lustre: server umount lustre-OST0000 complete [ 2250.210091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2250.230208] LustreError: Skipped 1 previous similar message [ 2250.235968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.261246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.280336] Lustre: server umount lustre-MDT0000 complete [ 2255.761462] LustreError: 63488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863042 with bad export cookie 920476810311693776 [ 2255.763285] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2255.774185] LustreError: 63488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2256.067319] Lustre: server umount lustre-MDT0001 complete [ 2268.763824] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2272.156537] Key type lgssc unregistered [ 2272.582697] LNet: 67069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.594091] LNetError: 67069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.623735] LNet: Removed LNI 192.168.204.144@tcp [ 2273.570656] Key type .llcrypt unregistered [ 2273.576385] Key type ._llcrypt unregistered [ 2301.918195] Key type ._llcrypt registered [ 2301.921987] Key type .llcrypt registered [ 2302.009733] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2302.928039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2302.972178] alg: No test for adler32 (adler32-zlib) [ 2304.192785] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 2304.362387] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2306.087408] Key type lgssc registered [ 2307.508053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2319.591877] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 16:38:23 (1787863103) [ 2328.581627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2328.625614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2330.794240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.564104] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2340.327745] LustreError: 68775:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.356834] LustreError: 68775:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2345.452776] LustreError: 68776:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.512555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.778264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2350.661923] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2358.157431] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2364.598527] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2371.662201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2372.072391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.188242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2379.642176] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2388.501272] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2397.996041] Lustre: Failing over lustre-MDT0000 [ 2398.178467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2398.184890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.196333] LustreError: 68776:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.322442] Lustre: server umount lustre-MDT0000 complete [ 2401.245673] LustreError: 69831:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.767209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.786974] Lustre: Skipped 1 previous similar message [ 2406.337807] LustreError: 68770:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.358956] LustreError: 68770:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2407.910711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.020198] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2408.202155] Lustre: *** cfs_fail_loc=135, val=0*** [ 2408.346293] LustreError: 71861:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2408.362293] LustreError: 71861:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.144@tcp: cfg command failed: rc = -2 [ 2408.374729] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2408.374729] [ 2408.382571] LustreError: MGC192.168.204.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2408.401420] LustreError: 71850:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2408.433317] LustreError: 71850:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2408.457001] LustreError: 71850:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2408.464038] LustreError: 71850:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2408.486952] Lustre: server umount lustre-MDT0000 complete [ 2408.490020] LustreError: 71850:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2415.009395] LustreError: 68769:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.033867] LustreError: 68769:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 2415.758684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.887655] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2415.907568] LustreError: 68767:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2416.248285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.360387] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2416.569060] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2420.923612] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2421.267556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2421.315519] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2421.350778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2431.467154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.482673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2431.499429] Lustre: Skipped 1 previous similar message [ 2431.514749] Lustre: Skipped 1 previous similar message [ 2434.502782] Lustre: server umount lustre-OST0000 complete [ 2441.700143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2441.705751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.711256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2441.724393] Lustre: Skipped 1 previous similar message [ 2444.598153] Lustre: server umount lustre-MDT0000 complete [ 2446.822657] LustreError: 68771:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.841338] LustreError: 68771:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2448.424977] LustreError: 68756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863235 with bad export cookie 1819329257272114237 [ 2448.431176] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.441296] LustreError: 68756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2448.724619] Lustre: server umount lustre-MDT0001 complete [ 2460.344219] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2463.105170] Key type lgssc unregistered [ 2463.387128] LNet: 73644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.404284] LNetError: 73644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2463.427307] LNet: Removed LNI 192.168.204.144@tcp [ 2464.383801] Key type .llcrypt unregistered [ 2464.386071] Key type ._llcrypt unregistered [ 2488.183491] Key type ._llcrypt registered [ 2488.185315] Key type .llcrypt registered [ 2488.283759] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2489.255970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2489.451483] alg: No test for adler32 (adler32-zlib) [ 2490.525491] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 2490.762394] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2492.431179] Key type lgssc registered [ 2493.518272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2500.612827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2500.648115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.159756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.306895] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2511.331429] LustreError: 75057:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.359582] LustreError: 75057:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2515.188030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.559902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2520.926168] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2529.811387] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2536.598971] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2543.604569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2544.055182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.143880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2550.688629] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2557.892919] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2566.381515] Lustre: Failing over lustre-MDT0000 [ 2566.626775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2566.634994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.649615] LustreError: 75785:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2566.717855] Lustre: server umount lustre-MDT0000 complete [ 2569.689777] LustreError: 77326:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.727182] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.764059] Lustre: Skipped 1 previous similar message [ 2574.777138] LustreError: 75052:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.796286] LustreError: 75052:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2575.806483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.963460] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.087817] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2576.275229] LustreError: 78146:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2576.293387] LustreError: 78146:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.144@tcp: cfg command failed: rc = -9 [ 2576.309454] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2576.309454] [ 2576.332992] LustreError: MGC192.168.204.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2576.347597] LustreError: 78134:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2576.354730] LustreError: 78134:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2576.362200] LustreError: 78134:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2576.367550] LustreError: 78134:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2576.407907] Lustre: server umount lustre-MDT0000 complete [ 2576.410818] LustreError: 78134:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2579.905436] LustreError: 75053:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.919064] LustreError: 75053:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2582.693428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.811939] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.819173] LustreError: 75049:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2583.078243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.132270] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2585.022593] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2587.040377] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2588.161187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2588.185905] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2588.251264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2598.382604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2598.391514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2598.394019] Lustre: Skipped 1 previous similar message [ 2598.409511] Lustre: Skipped 1 previous similar message [ 2603.148158] Lustre: server umount lustre-OST0000 complete [ 2608.613799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.616289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2608.619275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2608.641409] Lustre: Skipped 1 previous similar message [ 2613.728307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2613.735724] Lustre: Skipped 2 previous similar messages [ 2613.920088] Lustre: server umount lustre-MDT0000 complete [ 2617.800673] LustreError: 75037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863404 with bad export cookie 12964790349122628021 [ 2617.808916] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2617.814397] LustreError: 75037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2618.214803] Lustre: server umount lustre-MDT0001 complete [ 2629.629818] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2632.276253] Key type lgssc unregistered [ 2632.569828] LNet: 79935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2632.580817] LNetError: 79935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2632.606454] LNet: Removed LNI 192.168.204.144@tcp [ 2633.484738] Key type .llcrypt unregistered [ 2633.488250] Key type ._llcrypt unregistered [ 2646.725927] Key type ._llcrypt registered [ 2646.728470] Key type .llcrypt registered [ 2646.859189] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2647.255627] Key type .llcrypt unregistered [ 2647.262672] Key type ._llcrypt unregistered [ 2670.091130] Key type ._llcrypt registered [ 2670.092814] Key type .llcrypt registered [ 2670.247978] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2671.890195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2671.968076] alg: No test for adler32 (adler32-zlib) [ 2673.032474] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 2673.245390] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2674.903594] Key type lgssc registered [ 2675.968577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2682.498540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2682.521548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.969861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2687.735565] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2694.123728] LustreError: 81840:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.156016] LustreError: 81840:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2696.327101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2696.848416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2701.907148] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2709.182734] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2714.405936] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2721.245818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2721.632054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.550265] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2728.832853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2735.024943] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2744.292182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2744.295663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2744.305771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2744.309844] Lustre: Skipped 1 previous similar message [ 2744.325796] Lustre: Skipped 1 previous similar message [ 2749.414299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2749.422915] Lustre: Skipped 1 previous similar message [ 2749.593340] Lustre: server umount lustre-OST0000 complete [ 2754.535312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.543016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.549682] Lustre: Skipped 1 previous similar message [ 2759.601621] Lustre: server umount lustre-MDT0000 complete [ 2759.648754] LustreError: 81840:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2759.667026] LustreError: 81840:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2762.932913] LustreError: 81820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863549 with bad export cookie 5567165205633038738 [ 2762.944064] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2763.238315] Lustre: server umount lustre-MDT0001 complete [ 2774.712646] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2777.735580] Key type lgssc unregistered [ 2778.146894] LNet: 85379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2778.163359] LNetError: 85379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2778.176932] LNet: Removed LNI 192.168.204.144@tcp [ 2779.048119] Key type .llcrypt unregistered [ 2779.051331] Key type ._llcrypt unregistered [ 2802.497782] Key type ._llcrypt registered [ 2802.504406] Key type .llcrypt registered [ 2802.626763] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2803.750729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2803.769070] alg: No test for adler32 (adler32-zlib) [ 2804.786715] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 2805.010536] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2806.735884] Key type lgssc registered [ 2807.790874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2814.738405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2814.757673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.117421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.632322] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2825.696851] LustreError: 86791:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.713597] LustreError: 86791:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2828.455727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2828.912659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2833.337806] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2839.962392] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2845.481982] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2852.242881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2852.512933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.766192] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2860.724640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2863.962565] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2871.265958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2871.266429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2871.269761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2871.293746] Lustre: Skipped 1 previous similar message [ 2876.391675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2876.398290] Lustre: Skipped 1 previous similar message [ 2876.515716] Lustre: server umount lustre-OST0000 complete [ 2879.968690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2879.977151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2880.001434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2880.014566] Lustre: Skipped 1 previous similar message [ 2881.505259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.625965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2886.631520] Lustre: Skipped 3 previous similar messages [ 2891.745304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.754331] Lustre: Skipped 1 previous similar message [ 2894.303970] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2894.452304] Lustre: server umount lustre-MDT0000 complete [ 2896.864209] LustreError: 86786:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2896.885217] LustreError: 86786:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2897.957445] LustreError: 86772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863684 with bad export cookie 428630410900084285 [ 2897.966639] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2897.970641] LustreError: 86772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2898.442653] Lustre: server umount lustre-MDT0001 complete [ 2909.681415] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2912.413287] Key type lgssc unregistered [ 2912.674897] LNet: 90328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2912.686099] LNetError: 90328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2912.704886] LNet: Removed LNI 192.168.204.144@tcp [ 2913.420881] Key type .llcrypt unregistered [ 2913.424295] Key type ._llcrypt unregistered [ 2937.193606] Key type ._llcrypt registered [ 2937.199590] Key type .llcrypt registered [ 2937.316115] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2938.633495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2938.656520] alg: No test for adler32 (adler32-zlib) [ 2939.763152] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 2939.979777] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2941.671206] Key type lgssc registered [ 2942.823996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2950.790600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2950.808362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2952.136095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2956.624558] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2961.383862] LustreError: 91743:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.407095] LustreError: 91743:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2964.742465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.176503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2969.335224] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2976.540920] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2983.000602] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2989.893045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2989.997615] LustreError: 93572:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.214955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2996.278790] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2996.723962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3003.122984] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3014.129058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3014.137130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.160608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3014.395481] Lustre: server umount lustre-OST0000 complete [ 3021.795415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3021.804755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.841217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3024.149589] Lustre: server umount lustre-MDT0000 complete [ 3027.431620] LustreError: 91744:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.464658] LustreError: 91744:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3028.707572] LustreError: 94244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863815 with bad export cookie 2438498776442669425 [ 3028.714343] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3028.725025] LustreError: 94244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3039.455860] Lustre: server umount lustre-MDT0001 complete [ 3051.022710] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3054.504981] Key type lgssc unregistered [ 3054.848436] LNet: 95288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3054.853400] LNetError: 95288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3054.881662] LNet: Removed LNI 192.168.204.144@tcp [ 3056.189738] Key type .llcrypt unregistered [ 3056.192277] Key type ._llcrypt unregistered [ 3081.416748] Key type ._llcrypt registered [ 3081.419204] Key type .llcrypt registered [ 3081.502831] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3082.500241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3082.514661] alg: No test for adler32 (adler32-zlib) [ 3083.598964] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 3083.937195] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3085.617141] Key type lgssc registered [ 3087.053330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3095.066805] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3095.099438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.687437] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3115.808667] LustreError: 96673:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 3142.561070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3148.736590] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3157.308889] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3162.961792] Lustre: server umount lustre-OST0000 complete [ 3172.292039] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 16:52:36 (1787863956) [ 3179.795578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.223156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.562905] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3185.635440] LustreError: 98312:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.680298] LustreError: 98312:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3189.734751] LustreError: 98313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.026339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.389500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3198.583378] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3206.059212] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3212.337292] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3219.889975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3220.323213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.464627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3227.268313] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3234.410585] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3250.145792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.159096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3250.160277] Lustre: Skipped 1 previous similar message [ 3250.160317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3250.174984] LustreError: Skipped 1 previous similar message [ 3254.467820] Lustre: server umount lustre-OST0000 complete [ 3260.386546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.393019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3260.396392] Lustre: Skipped 1 previous similar message [ 3260.408343] Lustre: Skipped 2 previous similar messages [ 3265.505480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3265.521571] Lustre: Skipped 1 previous similar message [ 3269.601730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3269.610530] Lustre: Skipped 1 previous similar message [ 3272.159165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3272.310836] Lustre: server umount lustre-MDT0000 complete [ 3274.722237] LustreError: 98312:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3274.745879] LustreError: 98312:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3275.927598] LustreError: 98293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864062 with bad export cookie 5364349043897316486 [ 3275.932256] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3275.941640] LustreError: 98293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3276.506933] Lustre: server umount lustre-MDT0001 complete [ 3287.393073] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3290.297040] Key type lgssc unregistered [ 3290.625798] LNet: 102011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3290.635522] LNetError: 102011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3290.656889] LNet: Removed LNI 192.168.204.144@tcp [ 3291.586886] Key type .llcrypt unregistered [ 3291.589506] Key type ._llcrypt unregistered [ 3332.852663] Key type ._llcrypt registered [ 3332.857753] Key type .llcrypt registered [ 3332.991644] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3334.465288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3334.485080] alg: No test for adler32 (adler32-zlib) [ 3335.694813] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 3336.001203] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3337.743207] Key type lgssc registered [ 3339.009216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3347.449419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3347.471267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.938100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3353.746837] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3359.207945] LustreError: 104199:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.234018] LustreError: 104199:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3362.744386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.070688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3367.869793] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3375.729477] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3381.060864] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3387.570400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3387.808851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3393.445670] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3394.041545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3400.234243] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3409.377625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3409.386590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.397845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3409.398165] Lustre: Skipped 1 previous similar message [ 3409.408835] Lustre: Skipped 1 previous similar message [ 3414.506342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3414.514796] Lustre: Skipped 1 previous similar message [ 3415.509996] Lustre: server umount lustre-OST0000 complete [ 3419.616440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3419.622704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.634973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.645712] Lustre: Skipped 1 previous similar message [ 3419.667585] Lustre: Skipped 1 previous similar message [ 3424.737474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3424.744506] Lustre: Skipped 1 previous similar message [ 3425.744895] Lustre: server umount lustre-MDT0000 complete [ 3429.018415] LustreError: 104178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864215 with bad export cookie 16155824593184206530 [ 3429.021730] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3429.024544] LustreError: 104178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3429.332330] Lustre: server umount lustre-MDT0001 complete [ 3438.403593] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3441.652486] Key type lgssc unregistered [ 3441.923716] LNet: 107734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3441.937295] LNetError: 107734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3441.962240] LNet: Removed LNI 192.168.204.144@tcp [ 3442.962087] Key type .llcrypt unregistered [ 3442.963667] Key type ._llcrypt unregistered [ 3460.717831] Key type ._llcrypt registered [ 3460.723439] Key type .llcrypt registered [ 3460.835897] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3461.786436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3461.975356] alg: No test for adler32 (adler32-zlib) [ 3463.094833] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 3463.411259] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3465.111971] Key type lgssc registered [ 3466.256851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3474.026189] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3474.042876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3489.568990] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3494.687735] LustreError: 108993:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 3521.311330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.923782] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3531.808693] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3536.994374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.200616] LustreError: MGC192.168.204.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3537.208865] LustreError: 110016:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3537.213713] LustreError: 110016:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3537.221762] LustreError: 110016:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3537.314732] Lustre: server umount lustre-MDT0000 complete [ 3537.322764] LustreError: 110016:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3553.875950] Lustre: server umount lustre-OST0000 complete [ 3562.051979] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 3569.932997] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3601.713735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.915622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3601.949718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3602.008833] Lustre: lustre-MDT0000: new disk, initializing [ 3602.079747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3602.097724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3606.014125] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3615.221513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3615.346787] Lustre: 112807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3615.389281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3615.392493] Lustre: Skipped 1 previous similar message [ 3615.450457] Lustre: lustre-MDT0001: new disk, initializing [ 3615.508721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3615.547305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3615.559434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3619.712056] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3627.705889] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3632.790456] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3638.915597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3639.147270] Lustre: lustre-OST0000: new disk, initializing [ 3639.150808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3639.156893] Lustre: 114772:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3639.236204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.320400] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3649.156848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3649.170604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3649.335060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3654.006965] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3658.071741] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3658.256787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3662.502504] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3662.723890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3664.865847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3664.872215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.877597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3664.887629] Lustre: Skipped 1 previous similar message [ 3664.902447] Lustre: Skipped 1 previous similar message [ 3670.003203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3670.014789] Lustre: Skipped 1 previous similar message [ 3671.047047] Lustre: server umount lustre-OST0000 complete [ 3677.152637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3677.156162] LustreError: Skipped 1 previous similar message [ 3677.159225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.169804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3680.226558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3680.228079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3680.242824] Lustre: Skipped 1 previous similar message [ 3681.957157] Lustre: server umount lustre-MDT0000 complete [ 3685.709211] LustreError: 112800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864472 with bad export cookie 11592742501024705853 [ 3685.716861] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3685.718223] LustreError: 112800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3686.061645] Lustre: server umount lustre-MDT0001 complete [ 3693.434572] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 17:01:18 (1787864478) [ 3696.571402] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3712.165397] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 3722.025439] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3756.030938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.353271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3756.378650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3756.446402] Lustre: lustre-MDT0000: new disk, initializing [ 3756.509832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.532204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3761.266391] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3773.007624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3773.140426] Lustre: 119684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3773.224098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3773.229368] Lustre: Skipped 1 previous similar message [ 3773.306726] Lustre: lustre-MDT0001: new disk, initializing [ 3773.410069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3773.448486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3773.469064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3777.935536] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3787.961181] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3794.779766] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3801.106387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3801.312616] Lustre: lustre-OST0000: new disk, initializing [ 3801.329978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3801.352592] Lustre: 121657:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3801.424885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.130038] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3810.343193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3810.350884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3810.423642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3816.145708] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3821.451584] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3821.668927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3826.801652] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3827.026768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3830.752862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3830.767882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3830.783903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3835.320071] Lustre: server umount lustre-OST0000 complete [ 3839.968848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3839.978502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.996171] Lustre: Skipped 1 previous similar message [ 3840.002163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3840.008651] Lustre: Skipped 2 previous similar messages [ 3854.303510] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3854.529683] Lustre: server umount lustre-MDT0000 complete [ 3856.867539] LustreError: 119696:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3856.897393] LustreError: 119696:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 3857.723835] LustreError: 119675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864644 with bad export cookie 11592742501024706735 [ 3857.723835] LustreError: 121656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864644 with bad export cookie 11592742501024706735 [ 3857.724651] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3857.734272] LustreError: 121656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3857.990262] Lustre: server umount lustre-MDT0001 complete [ 3867.376515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3868.234515] LustreError: 123872:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3868.362316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.396087] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3873.761287] LustreError: 123873:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3877.864084] LustreError: 123872:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3881.821105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.285903] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3895.809804] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3902.193885] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3909.395495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3916.163916] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3923.944894] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3932.609426] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3937.763688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.768158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3937.779615] Lustre: Skipped 1 previous similar message [ 3937.794396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3937.803354] Lustre: Skipped 7 previous similar messages [ 3942.293956] Lustre: server umount lustre-OST0000 complete [ 3952.317347] Lustre: server umount lustre-MDT0000 complete [ 3953.120741] LustreError: 123868:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.132724] LustreError: 123868:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3955.929935] LustreError: 123854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864742 with bad export cookie 11592742501024707617 [ 3955.946222] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3955.948798] LustreError: 123854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3956.332274] Lustre: server umount lustre-MDT0001 complete [ 3967.569046] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3971.001560] Key type lgssc unregistered [ 3971.413138] LNet: 127507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3971.423637] LNetError: 127507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3971.455563] LNet: Removed LNI 192.168.204.144@tcp [ 3972.518436] Key type .llcrypt unregistered [ 3972.521233] Key type ._llcrypt unregistered [ 3991.475621] Key type ._llcrypt registered [ 3991.483606] Key type .llcrypt registered [ 3991.703899] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4006.967801] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4007.844259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4008.064114] alg: No test for adler32 (adler32-zlib) [ 4009.167511] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 4009.441068] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4011.119311] Key type lgssc registered [ 4012.330446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4048.325764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4048.336423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4049.580870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4049.624552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4049.788851] Lustre: lustre-MDT0000: new disk, initializing [ 4049.921406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4049.939634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4054.300899] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4065.570464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4065.737488] Lustre: 130662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4065.777971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4065.781484] Lustre: Skipped 1 previous similar message [ 4065.872561] Lustre: lustre-MDT0001: new disk, initializing [ 4065.991222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4066.036198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4066.045462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4070.984200] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4080.435423] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4085.853120] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4095.038452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4095.847326] Lustre: lustre-OST0000: new disk, initializing [ 4095.858885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4095.888128] Lustre: 132632:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4096.051679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.289611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4104.303189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4104.459469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4105.535945] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4114.335662] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4118.342343] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4118.534100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4122.792382] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4122.997831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4125.156774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4125.161143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4125.162260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4125.194667] Lustre: Skipped 1 previous similar message [ 4130.280220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4130.293631] Lustre: Skipped 2 previous similar messages [ 4135.396323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4135.404628] Lustre: Skipped 1 previous similar message [ 4138.975322] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4139.118763] Lustre: server umount lustre-OST0000 complete [ 4143.080952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4143.094075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.121294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4145.633313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.111555] Lustre: server umount lustre-MDT0000 complete [ 4152.928846] LustreError: 130654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864939 with bad export cookie 14956724955068973224 [ 4152.946720] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4152.952296] LustreError: 130654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4153.533978] Lustre: server umount lustre-MDT0001 complete [ 4162.209718] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 17:09:07 (1787864947) [ 4169.519912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4170.086949] LustreError: 135046:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.116539] LustreError: 135046:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4170.177316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4175.331814] LustreError: 135047:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4175.409157] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4179.424769] LustreError: 135046:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.645641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4183.932198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4188.354721] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4196.441905] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4203.052377] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4205.536696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.538159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.565266] Lustre: Skipped 1 previous similar message [ 4205.572267] Lustre: Skipped 2 previous similar messages [ 4211.384283] Lustre: server umount lustre-MDT0000 complete [ 4214.753623] LustreError: 135046:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4214.775372] LustreError: 135046:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4215.036081] LustreError: 135026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865001 with bad export cookie 14956724955068974106 [ 4215.040951] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4215.044309] LustreError: 135026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4215.351506] Lustre: server umount lustre-MDT0001 complete [ 4223.075590] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 17:10:08 (1787865008) [ 4229.180191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4244.831842] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4249.951570] LustreError: 137460:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 4275.679822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4281.710793] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4288.404285] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4290.551892] Lustre: server umount lustre-OST0000 complete [ 4298.548358] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 17:11:23 (1787865083) [ 4305.346488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4305.884578] LustreError: 138905:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4305.984690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4310.618456] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4316.137660] LustreError: 138905:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4316.162143] LustreError: 138905:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4319.119952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.255506] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4330.696764] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4336.287801] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4342.262417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4342.650154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4342.657018] Lustre: Skipped 1 previous similar message [ 4348.965942] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4355.952849] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4365.793812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4365.796267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4365.822954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4365.827714] Lustre: Skipped 3 previous similar messages [ 4368.586839] Lustre: server umount lustre-MDT0000 complete [ 4371.432781] LustreError: 139633:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.453426] LustreError: 139633:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4371.853514] LustreError: 138886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865158 with bad export cookie 14956724955068974771 [ 4371.861388] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4371.870320] LustreError: 138886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4372.126391] Lustre: server umount lustre-MDT0001 complete [ 4384.568200] Lustre: server umount lustre-OST0000 complete [ 4391.714996] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 17:12:56 (1787865176) [ 4397.592851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4401.989069] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4408.295291] LustreError: 142558:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.308574] LustreError: 142558:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4408.613949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.923084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4408.930057] Lustre: Skipped 1 previous similar message [ 4412.227105] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4418.274911] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4423.458379] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4428.835478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4434.189563] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4440.175694] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4443.522508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4444.436994] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4444.679482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4449.675163] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4449.940658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4453.860230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.864080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4453.874690] Lustre: Skipped 2 previous similar messages [ 4453.876590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4453.876596] Lustre: Skipped 3 previous similar messages [ 4458.218543] Lustre: server umount lustre-OST0000 complete [ 4468.053789] Lustre: server umount lustre-MDT0000 complete [ 4472.289874] LustreError: 145870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865259 with bad export cookie 14956724955068975842 [ 4472.295736] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4472.302533] LustreError: 145870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4472.730405] Lustre: server umount lustre-MDT0001 complete [ 4479.918814] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 17:14:25 (1787865265) [ 4485.899330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4501.472131] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4506.591937] LustreError: 146715:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 4536.945603] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4542.652822] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4549.303506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.881857] LustreError: 146740:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4549.904861] LustreError: 146740:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4549.957648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4549.963874] Lustre: Skipped 2 previous similar messages [ 4550.018522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4554.733812] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4563.027311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.980364] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4575.143432] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4580.347386] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4584.703659] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4584.902959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4589.173638] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4589.357192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4592.099025] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4592.126883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4592.131932] Lustre: Skipped 4 previous similar messages [ 4592.149606] Lustre: Skipped 5 previous similar messages [ 4597.488561] Lustre: server umount lustre-MDT0000 complete [ 4601.187214] LustreError: 146722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865387 with bad export cookie 14956724955068976864 [ 4601.194522] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4601.205838] LustreError: 146722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4601.671786] Lustre: server umount lustre-MDT0001 complete [ 4616.091627] Lustre: server umount lustre-OST0000 complete [ 4624.883850] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 17:16:50 (1787865410) [ 4631.990874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4647.647859] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4652.767526] LustreError: 150906:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 4683.695361] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4689.791337] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4695.891895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.237386] LustreError: 150931:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4696.264180] LustreError: 150931:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4696.395948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4700.605936] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4708.295331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4713.047188] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4721.119556] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4727.395456] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4734.553158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4734.752438] Lustre: lustre-OST0001: new disk, initializing [ 4734.759778] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4734.766187] Lustre: 153778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4741.735295] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4744.774126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4744.788742] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4744.940806] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4750.787877] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4755.697248] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4755.950126] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4760.050049] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4760.291168] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4763.104298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4763.114430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4763.132658] Lustre: Skipped 1 previous similar message [ 4763.140280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4763.149616] Lustre: Skipped 3 previous similar messages [ 4768.115794] Lustre: server umount lustre-OST0000 complete [ 4775.904797] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4775.926153] LustreError: Skipped 1 previous similar message [ 4778.448716] Lustre: server umount lustre-OST0001 complete [ 4797.407152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4797.614660] Lustre: server umount lustre-MDT0000 complete [ 4801.479487] LustreError: 152683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865588 with bad export cookie 14956724955068977816 [ 4801.480592] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4801.492318] LustreError: 152683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4802.082431] Lustre: server umount lustre-MDT0001 complete [ 4828.405831] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 17:20:13 (1787865613) [ 4829.822285] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4831.695738] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4834.016416] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 17:20:18 (1787865618) [ 4839.891218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.085815] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4840.105680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4840.233334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4840.243121] Lustre: Skipped 5 previous similar messages [ 4843.648723] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4854.360119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4854.452406] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4854.477178] Lustre: 157341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4859.093992] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4866.118950] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4871.397690] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4877.378628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4877.521949] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4883.250365] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4887.213886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4889.609140] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4893.932429] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4894.118525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4898.369112] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4898.584639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4902.371789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4902.381181] LustreError: Skipped 1 previous similar message [ 4902.388225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4902.405047] Lustre: Skipped 5 previous similar messages [ 4902.410398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4902.418078] Lustre: Skipped 11 previous similar messages [ 4906.292599] Lustre: server umount lustre-OST0000 complete [ 4926.009500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4932.119167] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4939.381620] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4942.639225] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4942.653126] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4942.654679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4944.336230] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4944.541705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4949.148858] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4949.380557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4954.935236] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98a907725000.ost_server_uuid 50 [ 4956.314767] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff98a907725000.ost_server_uuid in FULL state after 0 sec [ 4956.637318] LustreError: 158802:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.44@tcp) returned error from blocking AST (req@00000000b017f9b5 x1874712116583168 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000afff9901/0xcf90f61a3e1b5bf2 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.44@tcp remote: 0x27d12f95f083a016 expref: 5 pid: 157349 timeout: 5056 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4956.670258] LustreError: lustre-MDT0000: A client on nid 192.168.204.44@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4956.678424] LustreError: 157337:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.44@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000afff9901/0xcf90f61a3e1b5bf2 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.44@tcp remote: 0x27d12f95f083a016 expref: 6 pid: 157349 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4962.793547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4962.802271] LustreError: Skipped 1 previous similar message [ 4966.384424] Lustre: server umount lustre-OST0000 complete [ 4976.011834] Lustre: server umount lustre-MDT0000 complete [ 4978.658523] LustreError: 157353:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4978.673990] LustreError: 157353:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 4979.676615] LustreError: 157333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865766 with bad export cookie 14956724955068978999 [ 4979.679905] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4979.687929] LustreError: 157333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4986.214372] Lustre: server umount lustre-MDT0001 complete [ 4996.653928] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4999.588622] Key type lgssc unregistered [ 4999.902802] LNet: 163453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4999.911078] LNetError: 163453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4999.929510] LNet: Removed LNI 192.168.204.144@tcp [ 5000.935396] Key type .llcrypt unregistered [ 5000.937537] Key type ._llcrypt unregistered [ 5024.333333] Key type ._llcrypt registered [ 5024.337391] Key type .llcrypt registered [ 5024.462677] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5025.734150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5025.778132] alg: No test for adler32 (adler32-zlib) [ 5026.906235] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 5027.194656] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5028.872291] Key type lgssc registered [ 5030.219792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5038.102733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5038.130892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5039.598308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5044.496908] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5049.828206] LustreError: 164870:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.860966] LustreError: 164870:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5052.947252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5053.258213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5057.298897] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5066.566401] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5072.825990] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5080.032109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5080.515948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5084.661400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5086.479798] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5094.096526] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5103.998710] Lustre: Failing over lustre-MDT0000 [ 5104.611605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5104.621829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5104.637521] LustreError: 164869:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5106.191389] Lustre: server umount lustre-MDT0000 complete [ 5107.163352] LustreError: 166568:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.197791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.313989] LustreError: 167192:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.340369] LustreError: 167192:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 5118.407954] LustreError: 164850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.44@tcp arrived at 1787865905 with bad export cookie 14673986264952749065 [ 5118.419197] LustreError: 164850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5118.433680] LustreError: 165924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.456708] LustreError: 165924:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5123.361044] Lustre: 164181:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787865894/real 1787865894] req@00000000877b0e31 x1874713182760064/t0(0) o400->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787865910 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5123.391069] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5124.384738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5128.675075] LustreError: 165924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.688758] LustreError: 165924:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 5133.796175] LustreError: 164179:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005f74118c x1874713182765056/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5134.080875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5134.128399] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5138.249050] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5139.431778] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5139.437318] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5140.444641] Lustre: lustre-MDT0000: Denying connection for new client 319f2f20-a6d6-4fca-9048-0d54a287cbf6 (at 192.168.204.44@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5149.152892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5149.164880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.179234] Lustre: Skipped 1 previous similar message [ 5149.190554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5154.361982] Lustre: server umount lustre-OST0000 complete [ 5157.682216] LustreError: 168606:0:(ldlm_lib.c:3067:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5157.695713] Lustre: 167956:0:(ldlm_lib.c:2467:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5157.707920] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5157.749647] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5157.763183] LustreError: 167956:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d6f886f0 x1874713182779264/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5157.783979] LustreError: 167956:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5157.790420] LustreError: 167956:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5157.798432] Lustre: lustre-MDT0000: Recovery over after 0:18, of 2 clients 0 recovered and 2 were evicted. [ 5158.199635] Lustre: server umount lustre-MDT0000 complete [ 5159.909029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5159.925149] Lustre: Skipped 1 previous similar message [ 5159.930236] LustreError: 164865:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5159.939171] LustreError: 164865:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5161.200496] LustreError: 166714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865947 with bad export cookie 14673986264952749170 [ 5161.205369] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.216491] LustreError: 166714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5161.237145] LustreError: 164179:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5163.474838] Lustre: server umount lustre-MDT0001 complete [ 5173.483886] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5176.123368] Key type lgssc unregistered [ 5176.408742] LNet: 169361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5176.420995] LNetError: 169361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5176.439381] LNet: Removed LNI 192.168.204.144@tcp [ 5177.232722] Key type .llcrypt unregistered [ 5177.235760] Key type ._llcrypt unregistered [ 5199.866970] Key type ._llcrypt registered [ 5199.872388] Key type .llcrypt registered [ 5199.966848] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5200.946875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5200.959177] alg: No test for adler32 (adler32-zlib) [ 5201.990963] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 5202.173537] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5203.815259] Key type lgssc registered [ 5205.089906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5213.242476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5213.257139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.964038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5219.990971] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5224.481693] LustreError: 170772:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.511578] LustreError: 170772:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5229.236600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.626235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5234.098633] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5241.722973] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5246.934285] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5253.159689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5253.504040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5259.630069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5259.887682] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5268.028445] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5275.105386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.115965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5275.118223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5275.122036] Lustre: Skipped 1 previous similar message [ 5275.146877] Lustre: Skipped 1 previous similar message [ 5277.250597] Lustre: server umount lustre-OST0000 complete [ 5280.736910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5280.746892] LustreError: Skipped 1 previous similar message [ 5280.753191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5280.768966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.345244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.347831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.881834] Lustre: server umount lustre-MDT0000 complete [ 5290.470867] LustreError: 173350:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.503261] LustreError: 173350:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5290.521771] LustreError: 170753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866077 with bad export cookie 11237617336505834163 [ 5290.532517] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5290.540836] LustreError: 170753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5290.880669] Lustre: server umount lustre-MDT0001 complete [ 5303.464561] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5307.047746] Key type lgssc unregistered [ 5307.336707] LNet: 174307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5307.345128] LNetError: 174307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5307.366025] LNet: Removed LNI 192.168.204.144@tcp [ 5308.213535] Key type .llcrypt unregistered [ 5308.218696] Key type ._llcrypt unregistered [ 5341.560513] Key type ._llcrypt registered [ 5341.562944] Key type .llcrypt registered [ 5341.671615] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5342.785270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5342.932235] alg: No test for adler32 (adler32-zlib) [ 5343.954465] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 5344.372647] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5346.103179] Key type lgssc registered [ 5347.262446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5354.620656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5354.641320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.153915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5360.658916] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5365.735135] LustreError: 176014:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.749710] LustreError: 176014:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5368.291222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.595708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5373.125688] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5380.661601] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5386.681695] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5393.398372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5393.820648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5400.504833] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5401.028968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5407.372784] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5420.013410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5420.021190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5420.058051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5421.078167] Lustre: server umount lustre-OST0000 complete [ 5428.705698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.711566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5428.725375] Lustre: Skipped 1 previous similar message [ 5431.559134] Lustre: server umount lustre-MDT0000 complete [ 5433.829581] LustreError: 176015:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5433.850372] LustreError: 176015:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5435.191549] LustreError: 175995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866221 with bad export cookie 5953495835772977517 [ 5435.202241] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5435.209841] LustreError: 175995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5444.765600] Lustre: server umount lustre-MDT0001 complete [ 5454.786423] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5457.378825] Key type lgssc unregistered [ 5457.678226] LNet: 179551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5457.687315] LNetError: 179551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5457.707058] LNet: Removed LNI 192.168.204.144@tcp [ 5458.549707] Key type .llcrypt unregistered [ 5458.555691] Key type ._llcrypt unregistered [ 5477.836214] Key type ._llcrypt registered [ 5477.843257] Key type .llcrypt registered [ 5477.958236] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5478.911802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5479.055206] alg: No test for adler32 (adler32-zlib) [ 5480.183743] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 5480.474357] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5482.151621] Key type lgssc registered [ 5483.411530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5493.357719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5493.376642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5494.772429] Lustre: *** cfs_fail_loc=135, val=0*** [ 5494.875277] LustreError: 181054:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5494.882807] LustreError: 181054:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.144@tcp: cfg command failed: rc = -2 [ 5494.895284] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5494.895284] [ 5494.913146] LustreError: MGC192.168.204.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5494.938752] LustreError: 181015:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5494.945665] LustreError: 181015:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5494.954163] LustreError: 181015:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5494.965269] LustreError: 181015:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5495.056096] Lustre: server umount lustre-MDT0000 complete [ 5495.062059] LustreError: 181015:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5501.763322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.320439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5506.676329] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5512.672937] LustreError: 181425:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5512.691150] LustreError: 181425:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5514.958120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.344405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5520.283711] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5528.282166] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5534.044411] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5537.774246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5537.785250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5537.799902] Lustre: Skipped 1 previous similar message [ 5537.812248] Lustre: Skipped 1 previous similar message [ 5539.876240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5539.884611] Lustre: Skipped 1 previous similar message [ 5542.320224] Lustre: server umount lustre-MDT0000 complete [ 5544.930557] LustreError: 181425:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5544.948853] LustreError: 181425:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5545.650841] LustreError: 181407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866332 with bad export cookie 4923846243943964916 [ 5545.659517] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5545.662759] LustreError: 181407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5545.968641] Lustre: server umount lustre-MDT0001 complete [ 5555.468734] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5557.671751] Key type lgssc unregistered [ 5557.919425] LNet: 183751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5557.925574] LNetError: 183751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5557.939184] LNet: Removed LNI 192.168.204.144@tcp [ 5558.626582] Key type .llcrypt unregistered [ 5558.630302] Key type ._llcrypt unregistered [ 5578.365433] Key type ._llcrypt registered [ 5578.367194] Key type .llcrypt registered [ 5578.551843] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5579.005661] Key type .llcrypt unregistered [ 5579.012894] Key type ._llcrypt unregistered [ 5596.177505] Key type ._llcrypt registered [ 5596.180043] Key type .llcrypt registered [ 5596.283040] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5597.278132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5597.412879] alg: No test for adler32 (adler32-zlib) [ 5598.487218] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 5598.766813] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5600.463198] Key type lgssc registered [ 5601.664445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5609.248031] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5609.263622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5624.799613] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5629.919407] LustreError: 185887:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 5656.543319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5662.251207] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5669.551543] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5676.352610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5676.819662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5676.878166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5681.588793] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5682.161135] LustreError: 185913:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5682.185153] LustreError: 185913:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5687.271791] LustreError: 185912:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5690.711584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.114135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5695.915990] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5703.937387] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5709.778518] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5712.606600] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5727.201672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5727.209528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.233165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5728.229050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5728.234091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5728.235239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.257548] Lustre: Skipped 1 previous similar message [ 5730.621442] Lustre: server umount lustre-OST0000 complete [ 5737.440081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5737.446346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.466112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5741.047394] Lustre: server umount lustre-MDT0000 complete [ 5743.585535] LustreError: 186931:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5743.609078] LustreError: 186931:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5745.012724] LustreError: 185893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866531 with bad export cookie 1647367141042850298 [ 5745.018315] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5745.021366] LustreError: 185893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5745.337776] Lustre: server umount lustre-MDT0001 complete [ 5754.443179] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5757.367857] Key type lgssc unregistered [ 5757.676543] LNet: 189907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5757.690932] LNetError: 189907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5758.757264] LNet: Removed LNI 192.168.204.144@tcp [ 5759.727822] Key type .llcrypt unregistered [ 5759.733421] Key type ._llcrypt unregistered [ 5779.651187] Key type ._llcrypt registered [ 5779.653488] Key type .llcrypt registered [ 5779.764550] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5780.686379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5780.716275] alg: No test for adler32 (adler32-zlib) [ 5781.808947] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 5782.071452] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5783.793499] Key type lgssc registered [ 5784.762207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5791.436583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5791.446562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.793447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5796.634264] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5802.984757] LustreError: 191300:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5803.008960] LustreError: 191300:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5804.252661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5804.606423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5809.344958] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5817.686817] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5824.195676] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5832.096914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5832.598142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5837.770806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5839.793569] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5847.957231] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5852.880497] Lustre: Failing over lustre-MDT0000 [ 5853.334569] Lustre: server umount lustre-MDT0000 complete [ 5855.206860] LustreError: 192540:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.210284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.250628] Lustre: Skipped 2 previous similar messages [ 5860.283975] LustreError: 191296:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5860.308027] LustreError: 191296:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5865.399888] LustreError: 192540:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.432676] LustreError: 192540:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5870.547182] LustreError: 191295:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.591120] Lustre: 190610:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866641/real 1787866641] req@000000003fedf94f x1874713974433152/t0(0) o400->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787866657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5870.591196] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5870.616994] LustreError: 191295:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 5872.944448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5880.760766] LustreError: 191294:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5880.778696] LustreError: 191294:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 5880.806330] LustreError: 190608:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006f2295e7 x1874713974437760/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5881.047655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5881.106939] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5885.821389] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5885.884560] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5886.443902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5886.469249] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5886.540156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5890.888797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5896.912724] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5906.403794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5906.410200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5906.427771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5906.931860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5908.246644] Lustre: server umount lustre-OST0000 complete [ 5912.034686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5912.045310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5912.055290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5912.071178] Lustre: Skipped 2 previous similar messages [ 5917.160628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5917.173039] Lustre: Skipped 2 previous similar messages [ 5922.272222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5922.279452] Lustre: Skipped 1 previous similar message [ 5926.368289] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5926.537202] Lustre: server umount lustre-MDT0000 complete [ 5930.074651] LustreError: 191279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866716 with bad export cookie 7946055284587638767 [ 5930.076520] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5930.086481] LustreError: 191279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5930.530167] Lustre: server umount lustre-MDT0001 complete [ 5941.449723] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5944.750674] Key type lgssc unregistered [ 5945.091161] LNet: 196147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5945.107972] LNetError: 196147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5945.144220] LNet: Removed LNI 192.168.204.144@tcp [ 5946.056352] Key type .llcrypt unregistered [ 5946.060111] Key type ._llcrypt unregistered [ 5969.593646] Key type ._llcrypt registered [ 5969.595937] Key type .llcrypt registered [ 5969.750496] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5971.199058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5971.234264] alg: No test for adler32 (adler32-zlib) [ 5972.345252] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 5972.571125] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5974.208812] Key type lgssc registered [ 5975.257281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5982.980344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5983.003443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5984.594578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5990.051497] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5994.983183] LustreError: 197561:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5994.998027] LustreError: 197561:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5998.530131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.068830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6003.823954] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6010.770887] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6016.582175] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6022.176392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6022.485295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6028.569745] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6029.699096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6035.784706] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6038.609921] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6045.243675] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6063.274096] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6070.757622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.768958] Lustre: Skipped 1 previous similar message [ 6070.771175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6070.778531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6070.790858] Lustre: Skipped 1 previous similar message [ 6075.876760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6076.851599] Lustre: server umount lustre-OST0000 complete [ 6080.992961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6081.002692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6081.009698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6081.009706] Lustre: Skipped 1 previous similar message [ 6081.035747] Lustre: Skipped 1 previous similar message [ 6086.115614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6086.127420] Lustre: Skipped 2 previous similar messages [ 6086.770421] Lustre: server umount lustre-MDT0000 complete [ 6090.568893] LustreError: 197540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866877 with bad export cookie 17666173093550106607 [ 6090.575166] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6090.583077] LustreError: 197540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6090.879542] Lustre: server umount lustre-MDT0001 complete [ 6101.013689] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6104.428694] Key type lgssc unregistered [ 6104.786721] LNet: 201430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6104.798599] LNetError: 201430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6104.834661] LNet: Removed LNI 192.168.204.144@tcp [ 6105.948934] Key type .llcrypt unregistered [ 6105.958177] Key type ._llcrypt unregistered [ 6125.243546] Key type ._llcrypt registered [ 6125.247254] Key type .llcrypt registered [ 6125.456815] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6126.434995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6126.473879] alg: No test for adler32 (adler32-zlib) [ 6127.578916] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 6127.820784] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6129.575339] Key type lgssc registered [ 6131.311752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6141.250385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6141.315098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.374352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6149.311789] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6153.697324] LustreError: 202695:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.715110] LustreError: 202695:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6158.816306] LustreError: 202696:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.731936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6161.097835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6164.452863] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6170.732664] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6175.578473] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6181.510514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6181.807351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6186.663821] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6188.978025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6193.167378] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6198.715233] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6208.623875] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6217.736710] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6222.692677] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6235.107165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6235.110117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6235.113387] Lustre: Skipped 1 previous similar message [ 6235.125031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6238.926545] Lustre: server umount lustre-OST0000 complete [ 6242.786852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6242.793621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6242.820623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6242.830452] Lustre: Skipped 1 previous similar message [ 6245.345243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6245.346062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6250.467663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6250.472222] Lustre: Skipped 2 previous similar messages [ 6254.561369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6254.568503] Lustre: Skipped 1 previous similar message [ 6257.120314] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6257.610042] Lustre: server umount lustre-MDT0000 complete [ 6259.686841] LustreError: 202691:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6259.706082] LustreError: 202691:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6261.755886] LustreError: 202677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867048 with bad export cookie 6077835055136938485 [ 6261.759870] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6261.775953] LustreError: 202677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6262.111783] Lustre: server umount lustre-MDT0001 complete [ 6273.590191] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6276.429360] Key type lgssc unregistered [ 6276.738528] LNet: 207577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6276.752110] LNetError: 207577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6276.771439] LNet: Removed LNI 192.168.204.144@tcp [ 6277.847692] Key type .llcrypt unregistered [ 6277.851091] Key type ._llcrypt unregistered [ 6303.738926] Key type ._llcrypt registered [ 6303.741461] Key type .llcrypt registered [ 6303.842087] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6305.089693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6305.162722] alg: No test for adler32 (adler32-zlib) [ 6306.238937] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 6306.573682] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6308.311158] Key type lgssc registered [ 6309.455840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6317.337744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6317.392142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.999309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6324.206954] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6328.288976] LustreError: 208992:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6328.303890] LustreError: 208992:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6332.059576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6332.496564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6337.363544] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6344.272150] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6349.531778] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6356.123883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6356.454083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6362.546459] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6363.630248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6368.900852] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6383.073879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6383.081725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6383.096261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6384.106069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6384.111942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6384.134076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.193778] Lustre: server umount lustre-OST0000 complete [ 6393.824349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6393.832537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.842539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6393.848354] Lustre: Skipped 1 previous similar message [ 6396.621720] Lustre: server umount lustre-MDT0000 complete [ 6399.457452] LustreError: 208988:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.973736] LustreError: 208973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867186 with bad export cookie 11625178929080757048 [ 6399.976618] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6399.990805] LustreError: 208973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6400.280665] Lustre: server umount lustre-MDT0001 complete [ 6408.875372] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6411.117423] Key type lgssc unregistered [ 6411.337720] LNet: 212677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6411.346362] LNetError: 212677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6411.365885] LNet: Removed LNI 192.168.204.144@tcp [ 6411.891844] Key type .llcrypt unregistered [ 6411.896035] Key type ._llcrypt unregistered [ 6429.047851] Key type ._llcrypt registered [ 6429.049887] Key type .llcrypt registered [ 6429.129392] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6430.232777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6430.342362] alg: No test for adler32 (adler32-zlib) [ 6431.453634] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 6431.690837] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6433.327219] Key type lgssc registered [ 6434.288138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6442.077851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6442.093317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6443.799013] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.267404] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6454.243331] LustreError: 213942:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6454.268302] LustreError: 213942:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6459.369304] LustreError: 213941:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6459.820281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.489193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6466.546988] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6474.465368] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6480.110481] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6486.361455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6486.632927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6488.689262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6493.911662] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6501.476651] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6516.708059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6516.723663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.751257] Lustre: Skipped 1 previous similar message [ 6516.758049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6516.768186] Lustre: Skipped 1 previous similar message [ 6521.446934] Lustre: server umount lustre-OST0000 complete [ 6526.945359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.952449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.960045] Lustre: Skipped 1 previous similar message [ 6532.066462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6532.080996] Lustre: Skipped 1 previous similar message [ 6537.185280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6537.196800] Lustre: Skipped 1 previous similar message [ 6539.233487] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6539.373505] Lustre: server umount lustre-MDT0000 complete [ 6542.025063] LustreError: 213922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867328 with bad export cookie 14943186906663524484 [ 6542.038847] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6542.276945] Lustre: server umount lustre-MDT0001 complete [ 6551.529752] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6554.540671] Key type lgssc unregistered [ 6554.841600] LNet: 217631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6554.846363] LNetError: 217631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6554.869121] LNet: Removed LNI 192.168.204.144@tcp [ 6555.915217] Key type .llcrypt unregistered [ 6555.916841] Key type ._llcrypt unregistered [ 6572.222900] Key type ._llcrypt registered [ 6572.232212] Key type .llcrypt registered [ 6572.349546] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6573.221294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6573.289292] alg: No test for adler32 (adler32-zlib) [ 6574.406548] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 6574.729830] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6576.447815] Key type lgssc registered [ 6577.491550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6584.796381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6584.808176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.185935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6589.822655] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6595.554398] LustreError: 218876:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6595.564415] LustreError: 218876:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6597.553685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6597.991407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6602.484591] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6609.242941] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6614.481944] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6621.125688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6621.480239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6628.524555] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6630.750264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6637.861888] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6647.397013] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.476148] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6647.659650] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6654.190617] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6661.738134] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6663.424254] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6675.122767] Lustre: Permanently deactivating lustre-OST0001 [ 6675.146384] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6681.143684] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6688.317390] Lustre: Permanently reactivating lustre-OST0001 [ 6688.330542] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6688.335239] Lustre: Skipped 2 previous similar messages [ 6690.150719] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6690.167192] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6690.174483] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6690.190688] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6690.266551] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6699.495114] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6699.497638] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6699.516019] Lustre: Skipped 2 previous similar messages [ 6699.534595] Lustre: Skipped 1 previous similar message [ 6700.323305] Lustre: server umount lustre-OST0001 complete [ 6704.612769] LustreError: 220698:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.613788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6704.616819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6704.617617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6704.631789] LustreError: 220698:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6704.671394] Lustre: Skipped 1 previous similar message [ 6709.744620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6709.749689] Lustre: Skipped 2 previous similar messages [ 6709.752441] LustreError: 220696:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.779232] LustreError: 220696:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6710.404691] Lustre: server umount lustre-OST0000 complete [ 6714.849809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6714.859449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6714.868154] Lustre: Skipped 1 previous similar message [ 6714.890431] Lustre: Skipped 1 previous similar message [ 6719.974916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6719.989981] Lustre: Skipped 1 previous similar message [ 6720.579677] Lustre: server umount lustre-MDT0000 complete [ 6724.638045] LustreError: 218857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867511 with bad export cookie 140491511565803297 [ 6724.646712] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6724.651105] LustreError: 218857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6724.953683] Lustre: server umount lustre-MDT0001 complete [ 6736.460804] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6739.693205] Key type lgssc unregistered [ 6740.176228] LNet: 223926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6740.185447] LNetError: 223926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6740.226415] LNet: Removed LNI 192.168.204.144@tcp [ 6741.409539] Key type .llcrypt unregistered [ 6741.411635] Key type ._llcrypt unregistered [ 6780.471478] Key type ._llcrypt registered [ 6780.476540] Key type .llcrypt registered [ 6780.611452] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6781.868878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6781.922552] alg: No test for adler32 (adler32-zlib) [ 6783.049457] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 6783.324355] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6785.071297] Key type lgssc registered [ 6786.427592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6793.919685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6793.933773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6795.185094] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6795.225409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6795.502302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6800.018959] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6809.689720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6809.837210] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6809.897659] Lustre: 226129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6810.252718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6815.004943] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6822.516743] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6828.115237] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6836.580234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6836.699311] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6836.964785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6842.140576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6846.510286] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6855.939829] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6859.879319] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6866.273539] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6876.512269] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6884.652967] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6896.845603] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6896.858240] Lustre: Skipped 1 previous similar message [ 6918.928725] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6918.936398] Lustre: Skipped 2 previous similar messages [ 6954.535286] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6954.545826] Lustre: Skipped 4 previous similar messages [ 7022.808377] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7022.812199] Lustre: Skipped 7 previous similar messages [ 7031.785432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7031.785627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7031.795644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7031.800858] LustreError: Skipped 1 previous similar message [ 7031.817015] Lustre: Skipped 1 previous similar message [ 7036.433730] Lustre: server umount lustre-OST0000 complete [ 7040.993063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7041.003052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7041.016873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7041.020431] Lustre: Skipped 1 previous similar message [ 7042.016796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7042.021506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.440750] Lustre: server umount lustre-MDT0000 complete [ 7049.620484] LustreError: 226124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867836 with bad export cookie 16279012638519974363 [ 7049.634339] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7049.636341] LustreError: 226124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7050.090575] Lustre: server umount lustre-MDT0001 complete [ 7063.471491] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7067.633462] Key type lgssc unregistered [ 7067.817593] LNet: 230940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7067.833036] LNetError: 230940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7067.856683] LNet: Removed LNI 192.168.204.144@tcp [ 7068.698449] Key type .llcrypt unregistered [ 7068.700838] Key type ._llcrypt unregistered [ 7093.813626] Key type ._llcrypt registered [ 7093.816953] Key type .llcrypt registered [ 7093.925524] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7094.828557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7094.965201] alg: No test for adler32 (adler32-zlib) [ 7095.965736] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 7096.213704] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7097.895625] Key type lgssc registered [ 7098.982810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7105.659965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7105.676903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7107.117835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7112.239411] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7117.300406] LustreError: 232354:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7117.328388] LustreError: 232354:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7122.068568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7122.506697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7128.026320] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7135.949511] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7141.525893] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7147.300690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7147.629019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7152.704741] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7152.755045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7160.490724] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7177.370908] Lustre: server umount lustre-OST0000 complete [ 7183.841786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7183.844467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7183.850246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7183.859281] Lustre: Skipped 1 previous similar message [ 7183.866141] Lustre: Skipped 1 previous similar message [ 7188.974960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7188.985790] Lustre: Skipped 1 previous similar message [ 7191.177424] Lustre: server umount lustre-MDT0000 complete [ 7194.084540] LustreError: 232354:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7194.100583] LustreError: 232354:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7195.103242] Lustre: 231662:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867965/real 1787867965] req@00000000595eca03 x1874715352270464/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787867981 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7195.143189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7195.156247] LustreError: 232353:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7195.169175] LustreError: 232353:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7196.127161] Lustre: 231663:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867966/real 1787867966] req@00000000560a0c53 x1874715352270848/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787867982 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7196.344263] LustreError: 232334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867983 with bad export cookie 8618456536678223200 [ 7196.345692] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7196.365528] LustreError: 232334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7196.950652] Lustre: server umount lustre-MDT0001 complete [ 7209.394411] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7212.893602] Key type lgssc unregistered [ 7213.236529] LNet: 236043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7213.253316] LNetError: 236043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7214.314991] LNet: Removed LNI 192.168.204.144@tcp [ 7215.143476] Key type .llcrypt unregistered [ 7215.145533] Key type ._llcrypt unregistered [ 7240.114696] Key type ._llcrypt registered [ 7240.116412] Key type .llcrypt registered [ 7240.240527] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7242.039951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7242.326788] alg: No test for adler32 (adler32-zlib) [ 7243.425459] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 7243.738926] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7245.423273] Key type lgssc registered [ 7247.106421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7517.081439] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7520.419948] Key type lgssc unregistered [ 7520.861877] LNet: 238083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7520.872802] LNetError: 238083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7520.896732] LNet: Removed LNI 192.168.204.144@tcp [ 7522.042507] Key type .llcrypt unregistered [ 7522.051799] Key type ._llcrypt unregistered [ 7549.252701] Key type ._llcrypt registered [ 7549.256825] Key type .llcrypt registered [ 7549.411525] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7569.501673] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7570.728415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7570.802091] alg: No test for adler32 (adler32-zlib) [ 7571.914160] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 7572.151964] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7573.807191] Key type lgssc registered [ 7575.172812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7611.030983] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7289 sec ========== 18:06:35 (1787868395) [ 7612.576469] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:06:37 (1787868397) === [ 7616.474081] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:06:41 (1787868401) === [ 7657.644432] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7660.505701] Key type lgssc unregistered [ 7660.799910] LNet: 243322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7660.807463] LNetError: 243322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7660.818909] LNet: Removed LNI 192.168.204.144@tcp [ 7661.753531] Key type .llcrypt unregistered [ 7661.758556] Key type ._llcrypt unregistered