[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491637379 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002386] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010148] LSM: Security Framework initializing [ 0.011075] Yama: becoming mindful. [ 0.012048] SELinux: Initializing. [ 0.013078] *** VALIDATE selinux *** [ 0.021808] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026816] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027181] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029126] *** VALIDATE tmpfs *** [ 0.031461] *** VALIDATE proc *** [ 0.033076] *** VALIDATE cgroup *** [ 0.033951] *** VALIDATE cgroup2 *** [ 0.034272] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035177] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038012] Speculative Store Bypass: Vulnerable [ 0.040972] debug: unmapping init [mem 0xffffffff8a859000-0xffffffff8a860fff] [ 0.042190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043702] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046015] ... generic registers: 4 [ 0.047015] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049014] ... fixed-purpose events: 3 [ 0.050014] ... event mask: 000000070000000f [ 0.051316] rcu: Hierarchical SRCU implementation. [ 0.053505] smp: Bringing up secondary CPUs ... [ 0.054633] x86: Booting SMP configuration: [ 0.055038] .... node #0, CPUs: #1 #2 #3 [ 0.058236] smp: Brought up 1 node, 4 CPUs [ 0.060013] smpboot: Max logical packages: 1 [ 0.061021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148025] node 0 deferred pages initialised in 84ms [ 0.151143] devtmpfs: initialized [ 0.152222] x86/mm: Memory block size: 128MB [ 0.154980] gcov: version magic: 0x41383552 [ 0.156232] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159116] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161367] pinctrl core: initialized pinctrl subsystem [ 0.163173] [ 0.163752] ************************************************************* [ 0.166012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168012] ** ** [ 0.170015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173019] ** ** [ 0.175011] ** This means that this kernel is built to expose internal ** [ 0.177019] ** IOMMU data structures, which may compromise security on ** [ 0.179012] ** your system. ** [ 0.181016] ** ** [ 0.184015] ** If you see this message and you are not debugging the ** [ 0.186015] ** kernel, report this immediately to your vendor! ** [ 0.188013] ** ** [ 0.189012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.190018] ************************************************************* [ 0.193792] NET: Registered protocol family 16 [ 0.195416] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201100] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204102] cpuidle: using governor menu [ 0.205752] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208432] PCI: Using configuration type 1 for base access [ 0.209130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222274] cryptd: max_cpu_qlen set to 1000 [ 0.226309] ACPI: Added _OSI(Module Device) [ 0.227011] ACPI: Added _OSI(Processor Device) [ 0.228080] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235367] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243673] ACPI: Interpreter enabled [ 0.244055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246054] ACPI: Using IOAPIC for interrupt routing [ 0.247110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250454] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260993] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269111] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.274665] acpiphp: Slot [2] registered [ 0.276113] acpiphp: Slot [5] registered [ 0.277071] acpiphp: Slot [6] registered [ 0.278124] acpiphp: Slot [7] registered [ 0.280156] acpiphp: Slot [8] registered [ 0.281108] acpiphp: Slot [9] registered [ 0.282121] acpiphp: Slot [10] registered [ 0.284128] acpiphp: Slot [3] registered [ 0.285107] acpiphp: Slot [4] registered [ 0.287128] acpiphp: Slot [11] registered [ 0.289107] acpiphp: Slot [12] registered [ 0.290149] acpiphp: Slot [13] registered [ 0.291100] acpiphp: Slot [14] registered [ 0.293170] acpiphp: Slot [15] registered [ 0.295099] acpiphp: Slot [16] registered [ 0.297115] acpiphp: Slot [17] registered [ 0.298075] acpiphp: Slot [18] registered [ 0.299104] acpiphp: Slot [19] registered [ 0.301108] acpiphp: Slot [20] registered [ 0.302007] acpiphp: Slot [21] registered [ 0.303140] acpiphp: Slot [22] registered [ 0.305101] acpiphp: Slot [23] registered [ 0.306087] acpiphp: Slot [24] registered [ 0.306969] acpiphp: Slot [25] registered [ 0.309105] acpiphp: Slot [26] registered [ 0.310106] acpiphp: Slot [27] registered [ 0.311205] acpiphp: Slot [28] registered [ 0.312073] acpiphp: Slot [29] registered [ 0.313149] acpiphp: Slot [30] registered [ 0.315118] acpiphp: Slot [31] registered [ 0.317050] PCI host bridge to bus 0000:00 [ 0.319025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.321019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.329018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.331034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.334219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.336911] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.340406] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.351813] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.359083] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.361018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.368593] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.370943] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.374047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377776] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.383017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.398017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409427] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.420023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.426019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.441022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.450412] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.497018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.511359] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.518016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.524013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.544021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.556173] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.568019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.583018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.606019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.621185] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.631018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.640017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.663030] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.676649] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.686018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.696016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.718016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.730660] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.734401] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.736386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.739319] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.742252] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.748030] iommu: Default domain type: Passthrough [ 0.749445] SCSI subsystem initialized [ 0.751108] ACPI: bus type USB registered [ 0.753118] usbcore: registered new interface driver usbfs [ 0.754065] usbcore: registered new interface driver hub [ 0.756071] usbcore: registered new device driver usb [ 0.757169] pps_core: LinuxPPS API ver. 1 registered [ 0.759012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.761122] PTP clock support registered [ 0.763134] EDAC MC: Ver: 3.0.0 [ 0.766112] PCI: Using ACPI for IRQ routing [ 0.768832] NetLabel: Initializing [ 0.770012] NetLabel: domain hash size = 128 [ 0.772012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.774091] NetLabel: unlabeled traffic allowed by default [ 0.776122] vgaarb: loaded [ 0.778304] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.781016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.790004] clocksource: Switched to clocksource kvm-clock [ 0.897589] VFS: Disk quotas dquot_6.6.0 [ 0.900191] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.903376] *** VALIDATE ramfs *** [ 0.906043] *** VALIDATE hugetlbfs *** [ 0.908982] pnp: PnP ACPI init [ 0.911947] pnp: PnP ACPI: found 6 devices [ 0.935851] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.938451] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.941495] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.944518] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.947906] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.951375] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.955158] NET: Registered protocol family 2 [ 0.958194] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.964160] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.969066] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.975617] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.978767] TCP: Hash tables configured (established 65536 bind 65536) [ 0.980921] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.983200] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.985761] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.988299] NET: Registered protocol family 1 [ 0.990585] RPC: Registered named UNIX socket transport module. [ 0.992231] RPC: Registered udp transport module. [ 0.993754] RPC: Registered tcp transport module. [ 0.994696] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.996302] NET: Registered protocol family 44 [ 0.997942] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.000216] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.002395] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.005064] PCI: CLS 0 bytes, default 64 [ 1.006492] Unpacking initramfs... [ 2.414923] debug: unmapping init [mem 0xffff9e4efcc54000-0xffff9e4efffbffff] [ 2.419348] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.421589] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.424658] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.918093] Initialise system trusted keyrings [ 2.919934] Key type blacklist registered [ 2.921852] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.930730] zbud: loaded [ 2.933627] *** VALIDATE nfs *** [ 2.934875] *** VALIDATE nfs4 *** [ 2.936621] pstore: using deflate compression [ 2.939935] Platform Keyring initialized [ 3.049076] NET: Registered protocol family 38 [ 3.052990] Key type asymmetric registered [ 3.054922] Asymmetric key parser 'x509' registered [ 3.057153] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.060883] io scheduler mq-deadline registered [ 3.063139] io scheduler kyber registered [ 3.065227] io scheduler bfq registered [ 3.066733] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.069441] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.072257] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.075275] ACPI: Power Button [PWRF] [ 3.080931] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.088290] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.102265] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.109751] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.122763] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.151229] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.181640] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.187042] Non-volatile memory driver v1.3 [ 3.188418] Linux agpgart interface v0.103 [ 3.218321] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.221976] vda: detected capacity change from 0 to 76681216 [ 3.237068] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.240122] vdb: detected capacity change from 0 to 1073741824 [ 3.257117] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.260460] vdc: detected capacity change from 0 to 2621440000 [ 3.274912] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.277330] vdd: detected capacity change from 0 to 2621440000 [ 3.288932] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.291524] vde: detected capacity change from 0 to 4294967296 [ 3.306923] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.309083] vdf: detected capacity change from 0 to 4294967296 [ 3.315932] libphy: Fixed MDIO Bus: probed [ 3.350278] usbcore: registered new interface driver usbserial_generic [ 3.353171] usbserial: USB Serial support registered for generic [ 3.361139] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.373483] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.382220] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.388944] mousedev: PS/2 mouse device common for all mice [ 3.394713] rtc_cmos 00:05: RTC can wake from S4 [ 3.404147] rtc_cmos 00:05: registered as rtc0 [ 3.407344] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.409790] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.409836] intel_pstate: CPU model not supported [ 3.420077] hid: raw HID events driver (C) Jiri Kosina [ 3.430254] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.431826] usbcore: registered new interface driver usbhid [ 3.435962] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.440620] usbhid: USB HID core driver [ 3.450474] drop_monitor: Initializing network drop monitor service [ 3.453106] Initializing XFRM netlink socket [ 3.455272] NET: Registered protocol family 10 [ 3.458518] Segment Routing with IPv6 [ 3.460269] NET: Registered protocol family 17 [ 3.462414] mpls_gso: MPLS GSO support [ 3.467924] RAS: Correctable Errors collector initialized. [ 3.470166] AVX version of gcm_enc/dec engaged. [ 3.471899] AES CTR mode by8 optimization enabled [ 3.562156] sched_clock: Marking stable (3562135225, 0)->(4491065768, -928930543) [ 3.569286] registered taskstats version 1 [ 3.572166] Loading compiled-in X.509 certificates [ 3.575032] zswap: loaded using pool lzo/zbud [ 3.604935] Key type big_key registered [ 3.679608] Key type encrypted registered [ 3.681103] ima: No TPM chip found, activating TPM-bypass! [ 3.687657] ima: Allocated hash algorithm: sha1 [ 3.694809] ima: No architecture policies found [ 3.701639] evm: Initialising EVM extended attributes: [ 3.705132] evm: security.selinux [ 3.711210] evm: security.ima [ 3.713695] evm: security.capability [ 3.718050] evm: HMAC attrs: 0x1 [ 3.739200] rtc_cmos 00:05: setting system clock to 2026-09-06 09:07:23 UTC (1788685643) [ 3.764749] debug: unmapping init [mem 0xffffffff8b803000-0xffffffff8b9fffff] [ 3.778634] debug: unmapping init [mem 0xffffffff8a582000-0xffffffff8a858fff] [ 3.795980] Write protecting the kernel read-only data: 28672k [ 3.816144] debug: unmapping init [mem 0xffffffff88c03000-0xffffffff88dfffff] [ 3.823834] debug: unmapping init [mem 0xffffffff89514000-0xffffffff895fffff] [ 3.974741] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.999241] systemd[1]: Detected virtualization kvm. [ 4.005263] systemd[1]: Detected architecture x86-64. [ 4.012792] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.090780] systemd[1]: No hostname configured. [ 4.096532] systemd[1]: Set hostname to . [ 4.103989] random: systemd: uninitialized urandom read (16 bytes read) [ 4.108634] systemd[1]: Initializing machine ID from random generator. [ 4.312662] random: ln: uninitialized urandom read (6 bytes read) [ 4.659592] random: systemd: uninitialized urandom read (16 bytes read) [ 4.665735] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.681387] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.707162] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.083193] device-mapper: uevent: version 1.0.3 [ 7.091991] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.936503] virtio_net virtio0 ens2: renamed from eth0 [ 10.106979] random: fast init done [ 10.245996] scsi host0: ata_piix [ 10.429913] scsi host1: ata_piix [ 10.437658] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.451195] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.256473] random: crng init done [ 16.258992] random: 7 urandom warning(s) missed due to ratelimiting [ 18.771948] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.756834] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.308510] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.379047] SELinux: Disabled at runtime. [ 25.550841] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.582690] systemd[1]: Detected virtualization kvm. [ 25.584398] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.512567] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.517349] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.526315] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.531402] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.536820] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.551432] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.568834] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ 28.245681] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 29.194579] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.810887] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.898220] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.590249] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.681392] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 36.388396] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.277661] NFS: Registering the id_resolver key type [ 37.283837] Key type id_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.311279] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg401-server login: [ 53.188009] hrtimer: interrupt took 4000121 ns [ 96.459661] libcfs: loading out-of-tree module taints kernel. [ 96.567465] Key type ._llcrypt registered [ 96.571078] Key type .llcrypt registered [ 96.739793] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 119.387302] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 121.991214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 122.030890] alg: No test for adler32 (adler32-zlib) [ 123.590604] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 124.554632] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 126.404604] Key type lgssc registered [ 128.685495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 153.144519] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 199.560935] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 213.195947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 213.256171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.614957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 214.671427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 214.844913] Lustre: lustre-MDT0000: new disk, initializing [ 214.963909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.990549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 219.842337] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.787587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 232.862872] Lustre: 6523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 232.880787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 232.885519] Lustre: Skipped 1 previous similar message [ 232.963733] Lustre: lustre-MDT0001: new disk, initializing [ 233.040651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.095087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 233.111001] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 238.171563] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.429162] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 253.909357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.358617] Lustre: lustre-OST0000: new disk, initializing [ 254.361969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 254.366737] Lustre: 8462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.433193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.643873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 259.655875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.754566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 261.289793] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.693056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 276.834123] Lustre: lustre-OST0001: new disk, initializing [ 276.838206] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 276.846904] Lustre: 9535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 276.927162] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 283.716963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 283.729492] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 283.817790] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 283.956742] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 296.778711] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 306.313356] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 313.557762] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing check_logdir /tmp/testlogs/ [ 318.566710] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing yml_node [ 323.350814] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 325.739980] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 328.078475] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 329.731214] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 05:12:47 EDT 2026 [ 347.642718] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 348.938536] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 365.543202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.545751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.559500] Lustre: Skipped 3 previous similar messages [ 365.583781] Lustre: Skipped 3 previous similar messages [ 370.656557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 370.660748] Lustre: Skipped 1 previous similar message [ 371.771581] Lustre: server umount lustre-MDT0000 complete [ 375.940020] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686015 with bad export cookie 1438353480186275302 [ 375.944769] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.955485] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 380.905968] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.916479] Lustre: Skipped 1 previous similar message [ 380.925786] LustreError: 8470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.931391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 380.947456] LustreError: 8470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 380.962828] Lustre: Skipped 2 previous similar messages [ 382.443331] Lustre: server umount lustre-MDT0001 complete [ 387.418121] Lustre: server umount lustre-OST0000 complete [ 392.018411] Lustre: server umount lustre-OST0001 complete [ 400.196854] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 410.682816] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 451.309459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.582602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 451.621958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 451.692580] Lustre: lustre-MDT0000: new disk, initializing [ 451.773809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.792598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 456.204388] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 467.606361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.749702] Lustre: 15091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 467.835567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 467.846302] Lustre: Skipped 1 previous similar message [ 467.965731] Lustre: lustre-MDT0001: new disk, initializing [ 468.103804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 468.155824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 468.173663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 473.135591] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 482.840451] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 489.422544] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 496.889798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 497.172962] Lustre: lustre-OST0000: new disk, initializing [ 497.177402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 497.184286] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 497.271675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.403076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 498.413483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 498.588586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 504.359344] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 515.184748] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 520.817882] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 521.031710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 525.847306] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 526.066704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 529.399198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.400321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 529.403056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.413139] Lustre: Skipped 1 previous similar message [ 529.437510] LustreError: Skipped 1 previous similar message [ 534.455832] Lustre: server umount lustre-OST0000 complete [ 539.625277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 539.630898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.641115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.651043] Lustre: Skipped 1 previous similar message [ 539.662713] Lustre: Skipped 1 previous similar message [ 545.075357] Lustre: server umount lustre-MDT0000 complete [ 549.160926] LustreError: 17060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686188 with bad export cookie 1438353480186276835 [ 549.168165] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.170991] LustreError: 17060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 549.618711] Lustre: server umount lustre-MDT0001 complete [ 553.287193] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 05:16:31 (1788686191) [ 561.157763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.751279] LustreError: 19324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.840691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.266287] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.512800] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 571.363419] LustreError: 19324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.488693] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.432966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.224303] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 591.216935] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 597.352582] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 603.943469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 611.618261] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 619.612808] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 630.761693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 630.768374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.783409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 630.793892] Lustre: Skipped 3 previous similar messages [ 634.358277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 636.312639] Lustre: server umount lustre-OST0000 complete [ 644.576682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 644.585994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.606830] Lustre: Skipped 2 previous similar messages [ 646.972484] Lustre: server umount lustre-MDT0000 complete [ 649.701889] LustreError: 19324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 649.725925] LustreError: 19324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 650.711496] LustreError: 19307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686290 with bad export cookie 1438353480186277717 [ 650.712979] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 650.723901] LustreError: 19307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 651.283904] Lustre: server umount lustre-MDT0001 complete [ 662.889504] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 665.834573] Key type lgssc unregistered [ 666.124228] LNet: 22856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.138812] LNetError: 22856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.153487] LNet: Removed LNI 192.168.204.101@tcp [ 667.351187] Key type .llcrypt unregistered [ 667.353874] Key type ._llcrypt unregistered [ 690.140297] Key type ._llcrypt registered [ 690.142486] Key type .llcrypt registered [ 690.259871] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 692.085386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 692.119942] alg: No test for adler32 (adler32-zlib) [ 693.678716] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 693.858733] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 695.528168] Key type lgssc registered [ 697.887161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 708.683085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 708.725696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.468242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.797912] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 720.871274] LustreError: 24265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.897023] LustreError: 24265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 723.962377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.279037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 728.653746] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 736.681580] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 742.998946] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 750.346368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 750.647250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 755.764097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 756.937899] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 763.501594] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 779.235361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.237344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 779.278642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 784.360587] LustreError: 26089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 784.384521] LustreError: 26089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 784.561707] Lustre: server umount lustre-OST0000 complete [ 789.481400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 789.491324] Lustre: Skipped 1 previous similar message [ 789.494344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 789.498066] Lustre: Skipped 2 previous similar messages [ 794.592627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 794.607922] Lustre: Skipped 1 previous similar message [ 794.718149] Lustre: server umount lustre-MDT0000 complete [ 798.763166] LustreError: 24245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686438 with bad export cookie 13309294222879859914 [ 798.773668] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 798.786703] LustreError: 24245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 799.165350] Lustre: server umount lustre-MDT0001 complete [ 808.473686] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 811.219931] Key type lgssc unregistered [ 811.583755] LNet: 28240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.594157] LNetError: 28240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.605103] LNet: Removed LNI 192.168.204.101@tcp [ 812.741546] Key type .llcrypt unregistered [ 812.743443] Key type ._llcrypt unregistered [ 843.538450] Key type ._llcrypt registered [ 843.541263] Key type .llcrypt registered [ 843.686043] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 844.805212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 844.851505] alg: No test for adler32 (adler32-zlib) [ 845.942675] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 846.187124] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 847.919378] Key type lgssc registered [ 848.794339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 856.842429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 856.875436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.374066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.255251] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 868.840056] LustreError: 29653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.864468] LustreError: 29653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 870.764466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.262367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 875.869996] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 883.583901] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 889.717443] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 902.570965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 902.951043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 904.021432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 909.083236] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 916.600516] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 924.641875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 924.652202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.677381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.672031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 925.673378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.673799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.696371] Lustre: Skipped 1 previous similar message [ 929.817477] Lustre: server umount lustre-OST0000 complete [ 935.914329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.917178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.935692] Lustre: Skipped 1 previous similar message [ 935.958026] Lustre: Skipped 1 previous similar message [ 939.740158] Lustre: server umount lustre-MDT0000 complete [ 941.032994] LustreError: 29654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 941.058835] LustreError: 29654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 943.578863] LustreError: 29632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686583 with bad export cookie 18218961591145309200 [ 943.597609] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 943.882493] Lustre: server umount lustre-MDT0001 complete [ 954.006449] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 956.584945] Key type lgssc unregistered [ 956.837793] LNet: 33530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.847896] LNetError: 33530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.862432] LNet: Removed LNI 192.168.204.101@tcp [ 957.646989] Key type .llcrypt unregistered [ 957.648967] Key type ._llcrypt unregistered [ 980.155229] Key type ._llcrypt registered [ 980.160023] Key type .llcrypt registered [ 980.272514] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 981.405221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 981.432309] alg: No test for adler32 (adler32-zlib) [ 982.451578] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 982.706271] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 984.375167] Key type lgssc registered [ 985.399872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 992.045165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 992.070968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.428895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.272432] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1004.000268] LustreError: 34937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.019561] LustreError: 34937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1005.152683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.639355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1009.773756] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1016.127605] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1020.864305] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1025.434507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1025.670171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.843764] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1036.414373] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1039.022194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1049.575453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1049.579540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.587956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1049.606885] Lustre: Skipped 1 previous similar message [ 1049.620717] Lustre: Skipped 1 previous similar message [ 1054.696721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1054.711383] Lustre: Skipped 1 previous similar message [ 1055.600828] Lustre: server umount lustre-OST0000 complete [ 1059.823953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.829919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1059.836497] Lustre: Skipped 1 previous similar message [ 1059.849722] Lustre: Skipped 1 previous similar message [ 1064.930098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1064.948440] Lustre: Skipped 1 previous similar message [ 1065.242620] Lustre: server umount lustre-MDT0000 complete [ 1069.097076] LustreError: 34917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686708 with bad export cookie 8671450687878551764 [ 1069.106642] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1069.110369] LustreError: 34917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1069.523516] Lustre: server umount lustre-MDT0001 complete [ 1078.763715] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1081.563511] Key type lgssc unregistered [ 1081.829207] LNet: 38492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.834422] LNetError: 38492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.847680] LNet: Removed LNI 192.168.204.101@tcp [ 1082.572689] Key type .llcrypt unregistered [ 1082.574880] Key type ._llcrypt unregistered [ 1104.065908] Key type ._llcrypt registered [ 1104.071477] Key type .llcrypt registered [ 1104.207543] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1105.435143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1105.550047] alg: No test for adler32 (adler32-zlib) [ 1106.613957] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 1106.967964] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1108.647183] Key type lgssc registered [ 1110.098541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1118.210759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1118.259856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.737244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.817755] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1129.957959] LustreError: 39904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.984642] LustreError: 39904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1132.675345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.038629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1138.226481] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1146.307436] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1151.661059] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1158.616391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1159.009325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1165.181912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1165.865388] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1173.131689] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1185.760627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1185.769966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1185.790696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1187.812074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1187.815947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1187.818135] Lustre: Skipped 1 previous similar message [ 1187.837065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.130641] Lustre: server umount lustre-OST0000 complete [ 1209.825911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1209.834893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.859531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1209.865118] Lustre: Skipped 1 previous similar message [ 1212.386571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1212.389226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.072891] Lustre: server umount lustre-MDT0000 complete [ 1217.505086] LustreError: 41743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1217.530884] LustreError: 41743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1218.459918] LustreError: 39886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686858 with bad export cookie 17626106526939168401 [ 1218.465955] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1218.488818] LustreError: 39886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1224.975243] Lustre: server umount lustre-MDT0001 complete [ 1237.425821] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1240.552436] Key type lgssc unregistered [ 1240.854914] LNet: 43613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.866942] LNetError: 43613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1240.880864] LNet: Removed LNI 192.168.204.101@tcp [ 1241.984617] Key type .llcrypt unregistered [ 1241.989586] Key type ._llcrypt unregistered [ 1265.974249] Key type ._llcrypt registered [ 1265.976979] Key type .llcrypt registered [ 1266.143689] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1267.279107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1267.315922] alg: No test for adler32 (adler32-zlib) [ 1268.306611] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 1268.604339] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1270.279262] Key type lgssc registered [ 1271.441638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1280.454773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1280.478418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.931826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.109964] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1292.256913] LustreError: 45023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.279708] LustreError: 45023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1294.845141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.409564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1301.350387] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1309.917065] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1316.947159] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1323.177488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1323.643815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1327.732048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1330.464348] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1337.414401] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1350.402582] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.1@tcp (stopping) [ 1351.649670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1351.657232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1351.663334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1351.676865] Lustre: Skipped 2 previous similar messages [ 1351.693732] Lustre: Skipped 2 previous similar messages [ 1353.919536] Lustre: server umount lustre-MDT0000 complete [ 1355.483757] LustreError: 45018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.775555] LustreError: 45022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.789787] LustreError: 45022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1357.820639] LustreError: 45002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686997 with bad export cookie 2652823440934172657 [ 1357.845686] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1360.602408] LustreError: 45019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.642892] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.1@tcp (stopping) [ 1361.891385] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.915899] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1362.923240] Lustre: Skipped 1 previous similar message [ 1365.726510] LustreError: 45017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.741157] LustreError: 45017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1368.033908] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1368.045489] Lustre: Skipped 1 previous similar message [ 1372.130085] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1372.393896] Lustre: server umount lustre-MDT0001 complete [ 1387.639035] Lustre: server umount lustre-OST0000 complete [ 1402.991486] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1405.973510] Key type lgssc unregistered [ 1406.355676] LNet: 48883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.362192] LNetError: 48883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.376879] LNet: Removed LNI 192.168.204.101@tcp [ 1407.334914] Key type .llcrypt unregistered [ 1407.337848] Key type ._llcrypt unregistered [ 1428.798597] Key type ._llcrypt registered [ 1428.805238] Key type .llcrypt registered [ 1428.928533] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1430.077167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1430.181835] alg: No test for adler32 (adler32-zlib) [ 1431.303973] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 1431.542768] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1433.223202] Key type lgssc registered [ 1434.434981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1442.277740] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1442.296249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1457.826422] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1463.007605] LustreError: 50247:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 1489.695174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1495.442670] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1501.766429] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1666.125059] Lustre: server umount lustre-OST0000 complete [ 1681.567321] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1684.892426] Key type lgssc unregistered [ 1685.245184] LNet: 51920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.257471] LNetError: 51920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.286407] LNet: Removed LNI 192.168.204.101@tcp [ 1686.419196] Key type .llcrypt unregistered [ 1686.423802] Key type ._llcrypt unregistered [ 1710.651472] Key type ._llcrypt registered [ 1710.653792] Key type .llcrypt registered [ 1710.747518] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1711.635854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1711.734246] alg: No test for adler32 (adler32-zlib) [ 1712.821635] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 1713.169387] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1714.880083] Key type lgssc registered [ 1716.038200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1723.303372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1723.323280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.785221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.721953] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1735.142554] LustreError: 53333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1735.170742] LustreError: 53333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1736.057022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1736.469849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1740.442046] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1746.327427] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1752.064250] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1758.957867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1759.383958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.502426] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1769.660197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1772.620229] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1775.075551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.084415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1775.094845] Lustre: Skipped 1 previous similar message [ 1775.104740] Lustre: Skipped 1 previous similar message [ 1776.976452] Lustre: server umount lustre-OST0000 complete [ 1782.752659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1782.761045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.776764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.313288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.316112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.952730] Lustre: server umount lustre-MDT0000 complete [ 1790.439308] LustreError: 53329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.461365] LustreError: 53329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1791.050538] LustreError: 53314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687430 with bad export cookie 15570947313759890311 [ 1791.068400] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1791.405628] Lustre: server umount lustre-MDT0001 complete [ 1804.513733] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1807.854548] Key type lgssc unregistered [ 1808.272706] LNet: 56863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.278207] LNetError: 56863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.296308] LNet: Removed LNI 192.168.204.101@tcp [ 1809.287555] Key type .llcrypt unregistered [ 1809.292680] Key type ._llcrypt unregistered [ 1831.391169] Key type ._llcrypt registered [ 1831.396099] Key type .llcrypt registered [ 1831.479357] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1832.269695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1832.367801] alg: No test for adler32 (adler32-zlib) [ 1833.491491] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 1833.770939] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1835.439271] Key type lgssc registered [ 1836.588802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1843.899583] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1843.937133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1859.551498] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1864.671516] LustreError: 58243:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 1891.359222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1896.980535] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1903.994938] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1910.587947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.109566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1911.183863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1915.282784] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1916.386073] LustreError: 58269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.421441] LustreError: 58269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1921.534062] LustreError: 58268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1924.367638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1924.763141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1928.985536] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1936.404666] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1941.929821] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1945.056635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1945.064625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.077852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1946.595165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.604939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1946.616245] Lustre: Skipped 1 previous similar message [ 1949.851616] Lustre: server umount lustre-OST0000 complete [ 1971.809549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.816479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1971.826153] Lustre: Skipped 1 previous similar message [ 1977.083380] Lustre: server umount lustre-MDT0000 complete [ 1977.315438] LustreError: 59283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.325026] LustreError: 59283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1980.246632] LustreError: 58250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687620 with bad export cookie 8154173998994761674 [ 1980.254698] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1980.262262] LustreError: 58250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1986.655869] Lustre: server umount lustre-MDT0001 complete [ 1996.838821] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1999.181204] Key type lgssc unregistered [ 1999.441790] LNet: 61985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.453683] LNetError: 61985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1999.481919] LNet: Removed LNI 192.168.204.101@tcp [ 2000.279961] Key type .llcrypt unregistered [ 2000.282248] Key type ._llcrypt unregistered [ 2022.836185] Key type ._llcrypt registered [ 2022.839303] Key type .llcrypt registered [ 2022.981741] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2024.129663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2024.226159] alg: No test for adler32 (adler32-zlib) [ 2025.276470] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 2025.640996] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2027.399631] Key type lgssc registered [ 2028.702058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2035.917673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2035.960459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.352823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.615067] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2047.972585] LustreError: 63395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.986787] LustreError: 63395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2048.995571] LustreError: 63394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.666523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.971730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2056.062919] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2064.394288] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2070.129081] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2077.986188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2078.586078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.728795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2086.131258] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2094.473659] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2114.528739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2114.536861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.556271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.584361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.597178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2116.604844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.698725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2121.705177] Lustre: Skipped 1 previous similar message [ 2126.303713] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2126.476689] Lustre: server umount lustre-OST0000 complete [ 2131.938425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.942116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2131.948381] Lustre: Skipped 1 previous similar message [ 2135.754569] Lustre: server umount lustre-MDT0000 complete [ 2137.057240] LustreError: 64123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.075672] LustreError: 64123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2138.982656] LustreError: 63376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687778 with bad export cookie 10840498976896911111 [ 2138.987508] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2138.992949] LustreError: 63376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2139.303478] Lustre: server umount lustre-MDT0001 complete [ 2148.829621] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2151.363806] Key type lgssc unregistered [ 2151.711396] LNet: 66950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.733950] LNetError: 66950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2151.750418] LNet: Removed LNI 192.168.204.101@tcp [ 2152.587145] Key type .llcrypt unregistered [ 2152.592452] Key type ._llcrypt unregistered [ 2173.309558] Key type ._llcrypt registered [ 2173.311341] Key type .llcrypt registered [ 2173.398910] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2174.101891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2174.167269] alg: No test for adler32 (adler32-zlib) [ 2175.206752] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 2175.451120] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2177.151228] Key type lgssc registered [ 2178.115364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2186.484918] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 05:43:44 (1788687824) [ 2193.044601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2193.070478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.571889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2198.702110] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2205.154140] LustreError: 68628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.178699] LustreError: 68628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2207.640953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.924909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2212.478625] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2219.292522] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2225.144439] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2232.257079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2232.715296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.348774] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2239.868026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2245.756547] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2251.756776] Lustre: Failing over lustre-MDT0000 [ 2251.987129] Lustre: server umount lustre-MDT0000 complete [ 2253.582969] LustreError: 68623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.314343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2254.333932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.332881] LustreError: 68628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.334658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.361773] LustreError: 68628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2255.370868] Lustre: Skipped 1 previous similar message [ 2258.652435] LustreError: 68623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.671587] LustreError: 68623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2260.373264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.531351] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.635756] Lustre: *** cfs_fail_loc=135, val=0*** [ 2260.695081] LustreError: 71714:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2260.714635] LustreError: 71714:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.101@tcp: cfg command failed: rc = -2 [ 2260.731456] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2260.731456] [ 2260.738905] LustreError: MGC192.168.204.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2260.751974] LustreError: 71703:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2260.755954] LustreError: 71703:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2260.761221] LustreError: 71703:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2260.766290] LustreError: 71703:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2260.785412] Lustre: server umount lustre-MDT0000 complete [ 2260.790438] LustreError: 71703:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2263.766652] LustreError: 68623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.778905] LustreError: 68623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2266.114058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.199910] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2266.426067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.481321] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2268.889663] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2270.360744] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2271.728594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2271.752920] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2271.797917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2277.345158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2277.352533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.367837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.906992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2279.918259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.928834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2281.959629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2281.966802] Lustre: Skipped 1 previous similar message [ 2283.415452] Lustre: server umount lustre-OST0000 complete [ 2287.076468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2287.077187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.086029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.107043] Lustre: Skipped 1 previous similar message [ 2292.193550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.197642] Lustre: Skipped 2 previous similar messages [ 2292.978884] Lustre: server umount lustre-MDT0000 complete [ 2295.892929] LustreError: 68609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687935 with bad export cookie 17824119972096928185 [ 2295.896281] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.906911] LustreError: 68609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2296.138417] Lustre: server umount lustre-MDT0001 complete [ 2303.619406] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2305.866698] Key type lgssc unregistered [ 2306.169693] LNet: 73490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.182541] LNetError: 73490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2306.207995] LNet: Removed LNI 192.168.204.101@tcp [ 2306.930475] Key type .llcrypt unregistered [ 2306.936855] Key type ._llcrypt unregistered [ 2330.333979] Key type ._llcrypt registered [ 2330.338944] Key type .llcrypt registered [ 2330.429392] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2331.345887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2331.414916] alg: No test for adler32 (adler32-zlib) [ 2332.457715] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 2332.757421] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2334.418100] Key type lgssc registered [ 2335.910478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2344.187339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2344.223388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.806791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.113221] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2356.193437] LustreError: 74902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.207854] LustreError: 74902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2358.418515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.691665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2363.909433] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2371.431276] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2376.431443] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2383.764856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2384.230304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.152833] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2390.400937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2396.896905] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2410.648835] Lustre: Failing over lustre-MDT0000 [ 2410.897806] Lustre: server umount lustre-MDT0000 complete [ 2412.739364] LustreError: 74897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.049753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.073272] LustreError: 74898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.084706] LustreError: 74898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2417.881105] LustreError: 76742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.032748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.126216] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2419.272985] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2419.391359] LustreError: 78012:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2419.399355] LustreError: 78012:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.101@tcp: cfg command failed: rc = -9 [ 2419.409357] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2419.409357] [ 2419.420659] LustreError: MGC192.168.204.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2419.440524] LustreError: 78000:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2419.447902] LustreError: 78000:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2419.463549] LustreError: 78000:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2419.470629] LustreError: 78000:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2419.503089] Lustre: server umount lustre-MDT0000 complete [ 2419.507363] LustreError: 78000:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2423.004365] LustreError: 76742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.024598] LustreError: 76742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2424.873889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.007190] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.330152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2425.370189] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2428.122130] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2430.074055] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2430.449345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2430.498885] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2430.549561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2439.649245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2439.655691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.666948] Lustre: Skipped 2 previous similar messages [ 2439.671382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2440.674929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.681374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2440.704458] Lustre: Skipped 1 previous similar message [ 2444.185556] Lustre: server umount lustre-OST0000 complete [ 2450.913149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.913156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2450.914049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.926828] Lustre: Skipped 1 previous similar message [ 2453.870577] Lustre: server umount lustre-MDT0000 complete [ 2456.041559] LustreError: 74902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.068158] LustreError: 74902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2457.660796] LustreError: 74882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688097 with bad export cookie 13082489202086505970 [ 2457.676537] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2457.678964] LustreError: 74882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2458.029991] Lustre: server umount lustre-MDT0001 complete [ 2468.382490] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2471.341073] Key type lgssc unregistered [ 2471.630979] LNet: 79791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2471.637973] LNetError: 79791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2471.649555] LNet: Removed LNI 192.168.204.101@tcp [ 2472.540570] Key type .llcrypt unregistered [ 2472.544931] Key type ._llcrypt unregistered [ 2486.090884] Key type ._llcrypt registered [ 2486.093827] Key type .llcrypt registered [ 2486.231626] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2486.569738] Key type .llcrypt unregistered [ 2486.572921] Key type ._llcrypt unregistered [ 2509.551949] Key type ._llcrypt registered [ 2509.555975] Key type .llcrypt registered [ 2509.678977] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2511.052310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2511.072264] alg: No test for adler32 (adler32-zlib) [ 2512.184903] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 2512.448900] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2514.223206] Key type lgssc registered [ 2515.265961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2523.251640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2523.283512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.706490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.751059] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2534.896992] LustreError: 81685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.922430] LustreError: 81685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2539.211637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.666355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2544.843373] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2552.855466] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2558.391867] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2565.725405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2566.305182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2570.417800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2573.238566] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2581.703068] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2596.327338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.331608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2596.339860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2596.357448] Lustre: Skipped 1 previous similar message [ 2596.368221] LustreError: Skipped 1 previous similar message [ 2599.187365] Lustre: server umount lustre-OST0000 complete [ 2606.048858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2606.060038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2606.078477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2606.088536] Lustre: Skipped 1 previous similar message [ 2611.681583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2611.687304] Lustre: Skipped 2 previous similar messages [ 2616.803514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2616.815959] Lustre: Skipped 1 previous similar message [ 2617.311144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2617.575514] Lustre: server umount lustre-MDT0000 complete [ 2620.822891] LustreError: 81665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688260 with bad export cookie 8104703888699659170 [ 2620.838363] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.070983] Lustre: server umount lustre-MDT0001 complete [ 2631.952651] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2635.410201] Key type lgssc unregistered [ 2635.757824] LNet: 85219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2635.763253] LNetError: 85219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2635.775354] LNet: Removed LNI 192.168.204.101@tcp [ 2636.321761] Key type .llcrypt unregistered [ 2636.326396] Key type ._llcrypt unregistered [ 2661.536920] Key type ._llcrypt registered [ 2661.539922] Key type .llcrypt registered [ 2661.651575] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2662.465419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2662.541041] alg: No test for adler32 (adler32-zlib) [ 2663.565966] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 2663.757675] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2665.391254] Key type lgssc registered [ 2666.791249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2675.490802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2675.514365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.208821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.348970] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2686.439244] LustreError: 86630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.455798] LustreError: 86630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2690.703897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.105533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2695.665990] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2702.727745] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2708.951568] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2717.658579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2718.238057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.335593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2725.321439] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2733.522341] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2740.715156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2740.721481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.726421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2740.732148] Lustre: Skipped 1 previous similar message [ 2740.744424] Lustre: Skipped 1 previous similar message [ 2742.194593] Lustre: server umount lustre-OST0000 complete [ 2747.367863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2747.380163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.408529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2750.945636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2750.951279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2752.513665] Lustre: server umount lustre-MDT0000 complete [ 2756.033537] LustreError: 86610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688395 with bad export cookie 8517480178269196232 [ 2756.035126] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2756.045423] LustreError: 86610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2756.414326] Lustre: server umount lustre-MDT0001 complete [ 2768.001937] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2770.919982] Key type lgssc unregistered [ 2771.244995] LNet: 90162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2771.268619] LNetError: 90162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2771.291517] LNet: Removed LNI 192.168.204.101@tcp [ 2772.330658] Key type .llcrypt unregistered [ 2772.336444] Key type ._llcrypt unregistered [ 2797.282737] Key type ._llcrypt registered [ 2797.285779] Key type .llcrypt registered [ 2797.412312] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2798.834563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2798.947388] alg: No test for adler32 (adler32-zlib) [ 2799.972542] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 2800.189065] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2801.855218] Key type lgssc registered [ 2803.008428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2811.966116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2811.992935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.491587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.852488] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2823.652587] LustreError: 91572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.662503] LustreError: 91572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2825.869597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.264924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2831.207578] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2838.043255] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2843.003278] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2849.639361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2849.980419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2855.966819] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2859.202142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2864.054708] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2879.973997] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2879.982320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.984874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2880.002899] Lustre: Skipped 1 previous similar message [ 2885.093674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2885.101931] Lustre: Skipped 1 previous similar message [ 2890.214832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2890.223320] Lustre: Skipped 1 previous similar message [ 2891.231204] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2891.363588] Lustre: server umount lustre-OST0000 complete [ 2895.333247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.350938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.367830] Lustre: Skipped 2 previous similar messages [ 2900.449460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2900.459631] Lustre: Skipped 1 previous similar message [ 2901.342224] Lustre: server umount lustre-MDT0000 complete [ 2904.834531] LustreError: 91552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688544 with bad export cookie 17613849219309858287 [ 2904.835835] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2904.846465] LustreError: 91552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2905.217083] Lustre: server umount lustre-MDT0001 complete [ 2917.585424] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2921.152075] Key type lgssc unregistered [ 2921.716763] LNet: 95128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2921.725455] LNetError: 95128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2922.790617] LNet: Removed LNI 192.168.204.101@tcp [ 2924.239601] Key type .llcrypt unregistered [ 2924.243921] Key type ._llcrypt unregistered [ 2949.477521] Key type ._llcrypt registered [ 2949.482959] Key type .llcrypt registered [ 2949.608205] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2950.720417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2950.814950] alg: No test for adler32 (adler32-zlib) [ 2951.932944] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 2952.210462] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2953.911963] Key type lgssc registered [ 2954.952426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2962.060176] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2962.080679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2977.695447] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2982.817043] LustreError: 96519:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 3009.504811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3017.304474] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3025.610879] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3031.785616] Lustre: server umount lustre-OST0000 complete [ 3040.517069] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 05:57:58 (1788688678) [ 3047.443454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.144209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.542096] LustreError: 98158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.582748] LustreError: 98158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3053.586167] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3057.641028] LustreError: 98157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.637471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.757052] LustreError: 98158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.124632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3067.365739] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3075.324702] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3081.427929] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3088.033983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3088.455869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3092.529483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3094.154157] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3101.165961] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3118.061823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3118.069175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3118.070277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3118.088727] Lustre: Skipped 1 previous similar message [ 3118.121380] Lustre: Skipped 1 previous similar message [ 3121.417525] Lustre: server umount lustre-OST0000 complete [ 3128.289405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.293482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3128.306920] Lustre: Skipped 1 previous similar message [ 3128.336356] Lustre: Skipped 1 previous similar message [ 3131.500207] Lustre: server umount lustre-MDT0000 complete [ 3133.409045] LustreError: 99678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.419594] LustreError: 99678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3135.216475] LustreError: 98140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688774 with bad export cookie 13623325777701102944 [ 3135.224280] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3135.230970] LustreError: 98140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3135.652335] Lustre: server umount lustre-MDT0001 complete [ 3148.062780] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3150.507882] Key type lgssc unregistered [ 3150.696594] LNet: 101851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3150.702330] LNetError: 101851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3151.716946] LNet: Removed LNI 192.168.204.101@tcp [ 3152.430559] Key type .llcrypt unregistered [ 3152.435259] Key type ._llcrypt unregistered [ 3189.429505] Key type ._llcrypt registered [ 3189.432275] Key type .llcrypt registered [ 3189.516804] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3190.754263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3190.785015] alg: No test for adler32 (adler32-zlib) [ 3191.828491] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 3192.118658] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3193.815179] Key type lgssc registered [ 3194.883907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3202.720512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3202.769527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3204.193086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3210.217699] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3213.284853] LustreError: 104012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.320327] LustreError: 104012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3218.404450] LustreError: 104013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3218.461894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.957190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3224.295213] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3233.100661] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3239.779743] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3246.673521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3246.989989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3249.009618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3253.305472] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3260.699629] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3273.194251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3273.206215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.223095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3273.707975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3273.713331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.732598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3277.054259] Lustre: server umount lustre-OST0000 complete [ 3283.942674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.950889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.953042] Lustre: Skipped 1 previous similar message [ 3283.981531] Lustre: Skipped 1 previous similar message [ 3289.056733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.064541] Lustre: Skipped 1 previous similar message [ 3294.177190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.182107] Lustre: Skipped 1 previous similar message [ 3295.199675] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3295.393626] Lustre: server umount lustre-MDT0000 complete [ 3299.266921] LustreError: 103993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688939 with bad export cookie 15492040221949074588 [ 3299.278046] LustreError: 103993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3299.279730] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3299.875969] Lustre: server umount lustre-MDT0001 complete [ 3309.568401] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3312.102906] Key type lgssc unregistered [ 3312.408790] LNet: 107559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3312.423945] LNetError: 107559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3312.439457] LNet: Removed LNI 192.168.204.101@tcp [ 3313.328536] Key type .llcrypt unregistered [ 3313.334266] Key type ._llcrypt unregistered [ 3330.614650] Key type ._llcrypt registered [ 3330.624917] Key type .llcrypt registered [ 3330.800952] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3331.729062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3331.744895] alg: No test for adler32 (adler32-zlib) [ 3332.850477] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 3333.181529] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3334.983139] Key type lgssc registered [ 3336.653914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3346.464897] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3346.525811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3362.208465] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3367.327571] LustreError: 108797:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 3394.015790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3399.926696] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3406.763780] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3412.977553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3413.255830] LustreError: MGC192.168.204.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3413.269598] LustreError: 109821:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3413.276582] LustreError: 109821:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3413.285728] LustreError: 109821:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3413.375500] Lustre: server umount lustre-MDT0000 complete [ 3413.378681] LustreError: 109821:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3429.456116] Lustre: server umount lustre-OST0000 complete [ 3438.872639] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 3448.675562] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3485.441565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.663221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3485.690721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3485.755918] Lustre: lustre-MDT0000: new disk, initializing [ 3485.834852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3485.857918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3490.929526] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3501.802289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.963832] Lustre: 112612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3502.005559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3502.013694] Lustre: Skipped 1 previous similar message [ 3502.079382] Lustre: lustre-MDT0001: new disk, initializing [ 3502.188940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3502.252834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3502.265714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3507.727335] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3518.227716] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3523.953984] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3531.375248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3531.747572] Lustre: lustre-OST0000: new disk, initializing [ 3531.755993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3531.781421] Lustre: 114583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3531.913260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3538.571082] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3541.045138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3541.050731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3541.192895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3547.724135] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3554.241686] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3554.510959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3560.412741] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3560.579876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3567.077468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3567.081176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.105871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3567.124624] Lustre: Skipped 1 previous similar message [ 3567.158441] Lustre: Skipped 1 previous similar message [ 3569.116777] Lustre: server umount lustre-OST0000 complete [ 3577.318602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3577.329564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3577.348417] Lustre: Skipped 1 previous similar message [ 3577.359857] Lustre: Skipped 1 previous similar message [ 3581.127833] Lustre: server umount lustre-MDT0000 complete [ 3585.469014] LustreError: 112605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689225 with bad export cookie 16130886778014797494 [ 3585.473632] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3585.480876] LustreError: 112605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3585.933692] Lustre: server umount lustre-MDT0001 complete [ 3596.051990] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 06:07:14 (1788689234) [ 3599.732383] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3616.036528] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 3627.425953] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3664.357490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3664.711096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3664.748252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3664.873541] Lustre: lustre-MDT0000: new disk, initializing [ 3665.002187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3665.027192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3670.529719] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3681.591262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3681.698986] Lustre: 119484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3681.771980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3681.775995] Lustre: Skipped 1 previous similar message [ 3681.849420] Lustre: lustre-MDT0001: new disk, initializing [ 3681.935717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3681.980546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3681.993448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3686.087749] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3695.990274] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3701.856812] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3708.989058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3709.298962] Lustre: lustre-OST0000: new disk, initializing [ 3709.305820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3709.311435] Lustre: 121452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3709.423094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3716.643862] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3718.238194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3718.248855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3718.419284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3726.377468] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3731.882972] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3732.148795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3737.883982] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3738.238014] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3744.234629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3744.241335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.276638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3744.288328] Lustre: Skipped 1 previous similar message [ 3746.642927] Lustre: server umount lustre-OST0000 complete [ 3753.956493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3753.963809] LustreError: Skipped 1 previous similar message [ 3753.968870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.981985] Lustre: Skipped 1 previous similar message [ 3753.988217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3756.420340] Lustre: server umount lustre-MDT0000 complete [ 3759.459535] LustreError: 120277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689399 with bad export cookie 16130886778014798383 [ 3759.466083] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3759.475872] LustreError: 120277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3760.046178] Lustre: server umount lustre-MDT0001 complete [ 3767.923620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3768.324330] LustreError: 123670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.339831] LustreError: 123670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3768.364878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3773.414027] LustreError: 123671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3774.026698] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3778.542564] LustreError: 123670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3783.653338] LustreError: 123671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3786.650424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3792.651985] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3800.994766] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3807.077509] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3815.092312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3821.143028] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3828.525765] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3845.184385] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3852.258692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3852.269117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3852.285123] Lustre: Skipped 1 previous similar message [ 3852.300554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3852.311339] Lustre: Skipped 2 previous similar messages [ 3855.546471] Lustre: server umount lustre-OST0000 complete [ 3864.546167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3864.555444] Lustre: Skipped 5 previous similar messages [ 3867.583063] Lustre: server umount lustre-MDT0000 complete [ 3869.669131] LustreError: 124399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3869.687873] LustreError: 124399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3871.260214] LustreError: 123651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689511 with bad export cookie 16130886778014799265 [ 3871.266136] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3871.273975] LustreError: 123651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3871.613851] Lustre: server umount lustre-MDT0001 complete [ 3884.311750] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3887.154652] Key type lgssc unregistered [ 3887.497743] LNet: 127321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3887.507876] LNetError: 127321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3887.537573] LNet: Removed LNI 192.168.204.101@tcp [ 3888.481353] Key type .llcrypt unregistered [ 3888.484563] Key type ._llcrypt unregistered [ 3906.480434] Key type ._llcrypt registered [ 3906.483905] Key type .llcrypt registered [ 3906.618288] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 3923.091492] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3923.990860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3924.158328] alg: No test for adler32 (adler32-zlib) [ 3925.288140] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 3925.551439] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3927.247968] Key type lgssc registered [ 3928.709578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3964.170616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3964.197955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3965.510502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3965.539673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3965.607715] Lustre: lustre-MDT0000: new disk, initializing [ 3965.696338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.729435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3970.678436] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3981.589931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3981.735371] Lustre: 130471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3981.822191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3981.830889] Lustre: Skipped 1 previous similar message [ 3981.937632] Lustre: lustre-MDT0001: new disk, initializing [ 3982.130110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3982.217909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3982.233527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3987.860912] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3997.464594] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4003.191797] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4010.427772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4010.785638] Lustre: lustre-OST0000: new disk, initializing [ 4010.791761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4010.797687] Lustre: 132441:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4010.873629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4016.204823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4016.215579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4016.316210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4017.088821] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4026.033419] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4031.055293] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4031.303952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4036.522475] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4036.736878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4042.213059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4042.213544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.217734] LustreError: Skipped 1 previous similar message [ 4042.219526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4042.242076] Lustre: Skipped 1 previous similar message [ 4045.125291] Lustre: server umount lustre-OST0000 complete [ 4052.458931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4052.483141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4052.492156] Lustre: Skipped 2 previous similar messages [ 4055.807687] Lustre: server umount lustre-MDT0000 complete [ 4059.825555] LustreError: 130462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689699 with bad export cookie 9152174049472765249 [ 4059.830975] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4059.839865] LustreError: 130462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4060.233310] Lustre: server umount lustre-MDT0001 complete [ 4068.356828] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 06:15:06 (1788689706) [ 4075.052724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.656238] LustreError: 134854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4075.674329] LustreError: 134854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4075.783966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.520748] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4081.129028] LustreError: 134855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4086.247932] LustreError: 134854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.402675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.874536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4094.316516] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4102.418896] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4110.045234] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4114.469685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.472281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4114.482393] Lustre: Skipped 2 previous similar messages [ 4118.912196] Lustre: server umount lustre-MDT0000 complete [ 4119.521928] LustreError: 135582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4119.546986] LustreError: 135582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4123.145653] LustreError: 135589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689762 with bad export cookie 9152174049472766131 [ 4123.157679] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4123.445965] Lustre: server umount lustre-MDT0001 complete [ 4132.720433] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 06:16:10 (1788689770) [ 4140.174784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4155.935796] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4161.055748] LustreError: 137268:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 4186.784077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4193.836377] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4201.305957] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4203.876809] Lustre: server umount lustre-OST0000 complete [ 4213.405588] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 06:17:31 (1788689851) [ 4220.429796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.946580] LustreError: 138714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.051291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4226.048388] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4230.629276] LustreError: 138714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.646876] LustreError: 138714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4234.274161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4239.250436] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4246.971547] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4253.053491] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4260.716500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4261.229706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4261.242168] Lustre: Skipped 1 previous similar message [ 4267.944906] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4275.270772] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4288.404771] Lustre: server umount lustre-MDT0000 complete [ 4291.047546] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.059018] LustreError: 138714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4291.079669] Lustre: Skipped 1 previous similar message [ 4291.081112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4291.095438] LustreError: 138714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4292.498846] LustreError: 138697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689932 with bad export cookie 9152174049472766796 [ 4292.504999] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4292.518225] LustreError: 138697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4292.826581] Lustre: server umount lustre-MDT0001 complete [ 4307.453816] Lustre: server umount lustre-OST0000 complete [ 4316.166419] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 06:19:14 (1788689954) [ 4323.194798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.746609] LustreError: 142387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4328.814343] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4339.046070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4339.474678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4339.480990] Lustre: Skipped 1 previous similar message [ 4345.031814] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4353.277461] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4359.259189] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4366.009638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4368.433384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4372.012958] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4378.856700] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4382.043393] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4382.191184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4385.471458] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4385.690360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4388.833382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4388.847111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.857556] Lustre: Skipped 1 previous similar message [ 4388.862600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4388.868099] Lustre: Skipped 1 previous similar message [ 4393.649263] Lustre: server umount lustre-OST0000 complete [ 4400.099578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4400.138116] Lustre: Skipped 1 previous similar message [ 4400.148895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4400.165024] Lustre: Skipped 3 previous similar messages [ 4409.320377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4409.323485] Lustre: Skipped 2 previous similar messages [ 4412.383167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4412.739420] Lustre: server umount lustre-MDT0000 complete [ 4414.432373] LustreError: 142382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4414.445103] LustreError: 142382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4417.134674] LustreError: 142367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788690056 with bad export cookie 9152174049472767860 [ 4417.151045] LustreError: 142367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4417.151522] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4417.485481] Lustre: server umount lustre-MDT0001 complete [ 4426.836505] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 06:21:05 (1788690065) [ 4433.194493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4448.927628] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4454.048731] LustreError: 146547:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 4479.839393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.844101] Lustre: Skipped 1 previous similar message [ 4487.173725] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4494.810432] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4501.268831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4501.939890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4508.228931] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4517.809352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4523.663684] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4531.303888] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4537.895370] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4542.530904] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4542.744974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4547.562598] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4547.881387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4554.225751] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4554.227328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4554.237148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4554.243796] Lustre: Skipped 2 previous similar messages [ 4554.252697] LustreError: Skipped 1 previous similar message [ 4556.012515] Lustre: server umount lustre-MDT0000 complete [ 4559.339569] LustreError: 148561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4559.355205] LustreError: 148561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4560.308753] LustreError: 146554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788690200 with bad export cookie 9152174049472768889 [ 4560.343929] LustreError: 146554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4560.373817] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.177118] Lustre: server umount lustre-MDT0001 complete [ 4576.161409] Lustre: server umount lustre-OST0000 complete [ 4585.065823] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 06:23:43 (1788690223) [ 4591.551254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4607.071831] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4612.193026] LustreError: 150740:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 4645.588916] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4654.951953] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4662.876580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4663.711814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4668.598838] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4677.525552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.718581] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4691.416117] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4697.729269] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4705.335385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4705.639196] Lustre: lustre-OST0001: new disk, initializing [ 4705.642404] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4705.649685] Lustre: 153615:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4710.969551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4710.984951] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4711.083408] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4712.623212] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4722.748098] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4727.719828] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4727.886328] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4733.524615] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4733.926591] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4736.488654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.492470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4736.510584] Lustre: Skipped 2 previous similar messages [ 4736.532424] Lustre: Skipped 3 previous similar messages [ 4742.281241] Lustre: server umount lustre-OST0000 complete [ 4752.841338] Lustre: server umount lustre-OST0001 complete [ 4762.874837] Lustre: server umount lustre-MDT0000 complete [ 4767.297457] LustreError: 150746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788690407 with bad export cookie 9152174049472769841 [ 4767.302719] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4767.306321] LustreError: 150746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4767.604417] Lustre: server umount lustre-MDT0001 complete [ 4793.894392] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 06:27:11 (1788690431) [ 4795.648729] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4797.460555] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4799.097655] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 06:27:17 (1788690437) [ 4806.425588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.784806] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4806.828507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4807.064748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4807.072916] Lustre: Skipped 6 previous similar messages [ 4811.667575] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4820.105696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4820.245144] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4820.281209] Lustre: 157176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4825.277318] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4831.527406] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4837.839970] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4846.815914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4846.993130] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4852.648436] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4855.985831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4860.842739] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4867.599197] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4867.877650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4873.153504] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4873.469447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4876.781340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4876.789638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4876.793832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4876.793839] Lustre: Skipped 10 previous similar messages [ 4876.839537] Lustre: Skipped 5 previous similar messages [ 4881.750334] Lustre: server umount lustre-OST0000 complete [ 4902.048227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4909.879362] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4917.614578] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4917.738981] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4917.756832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4917.763495] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4923.190273] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4923.531549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4928.792068] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4929.032390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4935.428747] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a1e49be0800.ost_server_uuid 50 [ 4936.895854] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a1e49be0800.ost_server_uuid in FULL state after 0 sec [ 4937.205134] LustreError: 157182:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.1@tcp) returned error from blocking AST (req@00000000919535d7 x1875576949575040 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000c8fb69fa/0x7f030dcec55f788b lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.1@tcp remote: 0xc2473251e379c338 expref: 5 pid: 158235 timeout: 5037 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4937.243174] LustreError: lustre-MDT0000: A client on nid 192.168.204.1@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4937.249108] LustreError: 157171:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.1@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000c8fb69fa/0x7f030dcec55f788b lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.1@tcp remote: 0xc2473251e379c338 expref: 6 pid: 158235 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4942.822768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4942.828819] LustreError: Skipped 1 previous similar message [ 4948.437929] Lustre: server umount lustre-OST0000 complete [ 4958.688357] LustreError: 157181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4958.718549] LustreError: 157181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 4958.750547] Lustre: server umount lustre-MDT0000 complete [ 4962.752476] LustreError: 159037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788690602 with bad export cookie 9152174049472771024 [ 4962.757466] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4962.764023] LustreError: 159037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4977.119543] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4977.332842] Lustre: server umount lustre-MDT0001 complete [ 4988.854195] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 4991.792945] Key type lgssc unregistered [ 4992.189967] LNet: 163291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4992.201023] LNetError: 163291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4992.215249] LNet: Removed LNI 192.168.204.101@tcp [ 4993.062685] Key type .llcrypt unregistered [ 4993.067783] Key type ._llcrypt unregistered [ 5017.111537] Key type ._llcrypt registered [ 5017.114416] Key type .llcrypt registered [ 5017.226137] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5018.766614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5018.783694] alg: No test for adler32 (adler32-zlib) [ 5019.889128] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 5020.164280] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 5021.839520] Key type lgssc registered [ 5023.092561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5031.028389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5031.052071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5032.370440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5036.455180] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5042.658067] LustreError: 164703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5042.670498] LustreError: 164703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5044.559832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5045.128805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5050.659722] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5058.429340] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5063.453567] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5068.435528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5068.751918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5073.742066] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5077.936664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5078.615433] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5083.244967] Lustre: Failing over lustre-MDT0000 [ 5083.549102] Lustre: server umount lustre-MDT0000 complete [ 5086.178965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5086.193114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5086.221934] LustreError: 165431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5088.228889] LustreError: 166310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5088.240210] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5093.348245] LustreError: 164697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5093.374706] LustreError: 164697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5095.263093] LustreError: 164683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.1@tcp arrived at 1788690735 with bad export cookie 5914082067791406772 [ 5095.286838] LustreError: 164683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5098.470656] LustreError: 166310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5098.485752] LustreError: 166310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5102.590670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5102.793948] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5103.134585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.200554] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5107.682879] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5108.200821] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5108.228852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5110.088532] Lustre: lustre-MDT0000: Denying connection for new client fb65eb52-6bf4-4406-8401-bfcd6643c302 (at 192.168.204.1@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5121.003494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5121.014587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5121.031709] Lustre: Skipped 1 previous similar message [ 5121.049369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5123.554502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5123.559320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5128.677666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5128.690576] Lustre: Skipped 2 previous similar messages [ 5133.279819] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5133.483600] Lustre: server umount lustre-OST0000 complete [ 5137.684701] LustreError: 168442:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5137.692111] Lustre: 167790:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5137.702801] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5137.729729] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5137.738810] LustreError: 167790:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006e6c05bd x1875578096496896/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5137.764097] LustreError: 167790:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5137.778279] LustreError: 167790:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5137.788372] Lustre: lustre-MDT0000: Recovery over after 0:29, of 2 clients 0 recovered and 2 were evicted. [ 5138.146626] Lustre: server umount lustre-MDT0000 complete [ 5138.914792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5138.951277] LustreError: 164698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5141.791780] LustreError: 164682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788690781 with bad export cookie 5914082067791406870 [ 5141.810562] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5141.812325] LustreError: 164682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5144.400178] Lustre: server umount lustre-MDT0001 complete [ 5154.854375] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 5157.828903] Key type lgssc unregistered [ 5158.254826] LNet: 169192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5158.280063] LNetError: 169192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5158.302370] LNet: Removed LNI 192.168.204.101@tcp [ 5159.477425] Key type .llcrypt unregistered [ 5159.485535] Key type ._llcrypt unregistered [ 5184.024879] Key type ._llcrypt registered [ 5184.029195] Key type .llcrypt registered [ 5184.112253] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5184.624251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5184.644982] alg: No test for adler32 (adler32-zlib) [ 5185.701494] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 5185.933219] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 5187.607473] Key type lgssc registered [ 5188.794920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5196.278777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5196.294361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5197.856535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5202.229520] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5208.033941] LustreError: 170600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.042533] LustreError: 170600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5210.212734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5210.562530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5215.193651] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5224.049389] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5230.609374] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5236.881040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5237.295146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.420774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5243.491481] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5251.234890] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5257.442540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5257.463088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5258.531896] Lustre: server umount lustre-OST0000 complete [ 5262.816922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5262.822815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5262.836908] Lustre: Skipped 1 previous similar message [ 5262.853221] Lustre: Skipped 2 previous similar messages [ 5267.937522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5267.944414] Lustre: Skipped 1 previous similar message [ 5268.490323] Lustre: server umount lustre-MDT0000 complete [ 5272.604202] LustreError: 170581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788690912 with bad export cookie 4535437267725759899 [ 5272.609948] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5272.619538] LustreError: 170581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5273.068629] Lustre: server umount lustre-MDT0001 complete [ 5284.432930] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 5287.017927] Key type lgssc unregistered [ 5287.296277] LNet: 174130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5287.301672] LNetError: 174130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5287.312899] LNet: Removed LNI 192.168.204.101@tcp [ 5288.153904] Key type .llcrypt unregistered [ 5288.157983] Key type ._llcrypt unregistered [ 5320.200854] Key type ._llcrypt registered [ 5320.207153] Key type .llcrypt registered [ 5320.324042] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5320.906078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5320.940149] alg: No test for adler32 (adler32-zlib) [ 5321.979935] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 5322.210776] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 5323.849604] Key type lgssc registered [ 5325.316074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5334.193268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5334.229373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5335.744466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5341.141469] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5345.251120] LustreError: 175828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5345.272617] LustreError: 175828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5350.369601] LustreError: 175829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5351.133446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5351.610891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5355.964300] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5364.989624] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5372.337905] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5379.837460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5380.367881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5381.447266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5388.320786] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5396.048800] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5407.209405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5407.210672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5407.211537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5407.246359] Lustre: Skipped 1 previous similar message [ 5407.282774] LustreError: Skipped 1 previous similar message [ 5409.726500] Lustre: server umount lustre-OST0000 complete [ 5412.833222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5412.840776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5412.855418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5412.860732] Lustre: Skipped 1 previous similar message [ 5417.445753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5417.465260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5422.563830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5422.572793] Lustre: Skipped 2 previous similar messages [ 5427.167764] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5427.347458] Lustre: server umount lustre-MDT0000 complete [ 5427.692349] LustreError: 178321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5427.710077] LustreError: 178321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5430.478388] LustreError: 175808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788691070 with bad export cookie 11934393612297484530 [ 5430.489311] LustreError: 175808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5430.490826] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5430.930097] Lustre: server umount lustre-MDT0001 complete [ 5441.652611] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 5444.502748] Key type lgssc unregistered [ 5444.814689] LNet: 179363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5444.820718] LNetError: 179363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5444.842734] LNet: Removed LNI 192.168.204.101@tcp [ 5445.827095] Key type .llcrypt unregistered [ 5445.829472] Key type ._llcrypt unregistered [ 5468.322948] Key type ._llcrypt registered [ 5468.329272] Key type .llcrypt registered [ 5468.503825] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5470.241851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5470.288889] alg: No test for adler32 (adler32-zlib) [ 5471.447974] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 5471.888306] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 5473.655294] Key type lgssc registered [ 5474.701207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5486.134934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5486.165934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5487.509872] Lustre: *** cfs_fail_loc=135, val=0*** [ 5487.565455] LustreError: 180884:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5487.569725] LustreError: 180884:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.101@tcp: cfg command failed: rc = -2 [ 5487.576461] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5487.576461] [ 5487.581097] LustreError: MGC192.168.204.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5487.588400] LustreError: 180845:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5487.593144] LustreError: 180845:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5487.597205] LustreError: 180845:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5487.601501] LustreError: 180845:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5487.641557] Lustre: server umount lustre-MDT0000 complete [ 5487.643202] LustreError: 180845:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5494.140254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5494.614597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5499.038287] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5504.995478] LustreError: 181256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5505.018348] LustreError: 181256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5508.179580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5508.462769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5513.178734] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5520.973099] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5527.455767] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5530.084241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5530.121541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5530.134158] Lustre: Skipped 1 previous similar message [ 5534.242297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5534.255169] Lustre: Skipped 1 previous similar message [ 5535.835286] Lustre: server umount lustre-MDT0000 complete [ 5539.309605] LustreError: 181985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5539.318794] LustreError: 181985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5539.876650] LustreError: 181238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788691179 with bad export cookie 446163816792972596 [ 5539.895894] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5539.899023] LustreError: 181238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5540.309085] Lustre: server umount lustre-MDT0001 complete [ 5552.126708] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 5555.263837] Key type lgssc unregistered [ 5555.584852] LNet: 183577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5555.591486] LNetError: 183577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5555.610427] LNet: Removed LNI 192.168.204.101@tcp [ 5556.554591] Key type .llcrypt unregistered [ 5556.560925] Key type ._llcrypt unregistered [ 5580.411375] Key type ._llcrypt registered [ 5580.413947] Key type .llcrypt registered [ 5580.533244] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 5580.853422] Key type .llcrypt unregistered [ 5580.856074] Key type ._llcrypt unregistered [ 5598.815506] Key type ._llcrypt registered [ 5598.818422] Key type .llcrypt registered [ 5598.933791] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5600.123972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5600.156275] alg: No test for adler32 (adler32-zlib) [ 5601.312342] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 5601.556647] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 5603.239204] Key type lgssc registered [ 5604.567135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5613.021646] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5613.081839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5628.639415] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5633.760605] LustreError: 185702:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 5660.447206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5666.844694] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5674.707918] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5682.384279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.885151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5683.007209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5687.881238] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5688.292079] LustreError: 185727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.322921] LustreError: 185727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5693.410326] LustreError: 185728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.451921] LustreError: 186742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5696.924372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.374464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5702.295447] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5709.460843] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5715.028424] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5717.713677] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5733.356138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5733.356431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5733.367093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5736.414693] Lustre: server umount lustre-OST0000 complete [ 5743.588845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.596808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5743.596937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5743.602542] Lustre: Skipped 2 previous similar messages [ 5743.605451] Lustre: Skipped 1 previous similar message [ 5748.704997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5748.711360] Lustre: Skipped 2 previous similar messages [ 5753.826020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5753.830303] Lustre: Skipped 1 previous similar message [ 5754.335134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5754.713494] Lustre: server umount lustre-MDT0000 complete [ 5758.526202] LustreError: 185708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788691398 with bad export cookie 3018200186057300655 [ 5758.532676] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5758.535013] LustreError: 185708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5758.797592] Lustre: server umount lustre-MDT0001 complete [ 5769.746478] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 5772.945248] Key type lgssc unregistered [ 5773.236897] LNet: 189717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5773.253578] LNetError: 189717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5773.273443] LNet: Removed LNI 192.168.204.101@tcp [ 5774.364781] Key type .llcrypt unregistered [ 5774.369423] Key type ._llcrypt unregistered [ 5799.696883] Key type ._llcrypt registered [ 5799.702897] Key type .llcrypt registered [ 5799.821553] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5800.937973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5801.014079] alg: No test for adler32 (adler32-zlib) [ 5802.197887] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 5802.502267] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 5804.267336] Key type lgssc registered [ 5805.599719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5814.258186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5814.284410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.852855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5820.765617] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5826.034151] LustreError: 191129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5826.060460] LustreError: 191129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5830.287501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.737363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5835.864294] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5843.722937] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5849.788565] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5856.842632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5857.287381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.436239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5863.294654] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5871.083882] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5877.095475] Lustre: Failing over lustre-MDT0000 [ 5879.031901] LustreError: 191126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.490632] Lustre: server umount lustre-MDT0000 complete [ 5880.803614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5880.823435] LustreError: 191126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5880.825828] Lustre: Skipped 2 previous similar messages [ 5880.857527] LustreError: 191126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5884.121039] LustreError: 193148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.241690] LustreError: 191124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.258390] LustreError: 191124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5897.183373] Lustre: 190441:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788691520/real 1788691520] req@00000000f45d4897 x1875578917512448/t0(0) o400->MGC192.168.204.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788691536 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5897.213610] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5898.261970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5899.479705] LustreError: 191125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5899.503824] LustreError: 191125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5906.401465] LustreError: 190437:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b958e68f x1875578917517696/t0(0) o250->MGC192.168.204.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5906.628780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5906.710991] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5910.754528] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5911.554108] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5912.050156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5912.094943] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5912.177569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5917.574342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5922.242130] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5932.005677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5932.020164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5932.036083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5933.449259] Lustre: server umount lustre-OST0000 complete [ 5937.634071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5937.634785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5937.646136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5937.646143] Lustre: Skipped 1 previous similar message [ 5937.679188] Lustre: Skipped 2 previous similar messages [ 5942.752818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5942.770779] Lustre: Skipped 2 previous similar messages [ 5943.387930] Lustre: server umount lustre-MDT0000 complete [ 5947.323968] LustreError: 191111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788691587 with bad export cookie 15214987237462586585 [ 5947.325675] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5947.336128] LustreError: 191111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5947.731112] Lustre: server umount lustre-MDT0001 complete [ 5959.863537] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 5962.993718] Key type lgssc unregistered [ 5963.339220] LNet: 195923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5963.347555] LNetError: 195923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5964.391646] LNet: Removed LNI 192.168.204.101@tcp [ 5965.578134] Key type .llcrypt unregistered [ 5965.579825] Key type ._llcrypt unregistered [ 5994.587619] Key type ._llcrypt registered [ 5994.593324] Key type .llcrypt registered [ 5994.754752] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5995.671839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5995.729725] alg: No test for adler32 (adler32-zlib) [ 5996.870807] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 5997.113719] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 5998.759395] Key type lgssc registered [ 5999.746542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6007.706478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6007.731291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6009.602906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6015.778367] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6020.081893] LustreError: 197333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6020.119305] LustreError: 197333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6025.186576] LustreError: 197334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.989316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.584627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6032.880316] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6040.118684] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6045.051480] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6050.876898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6051.184198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6054.260355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6056.254474] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6063.800554] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6067.659770] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6072.385731] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6086.888157] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6094.307026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6094.328219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6094.363824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6095.328515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6095.330649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6095.334828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6097.432198] Lustre: server umount lustre-OST0000 complete [ 6104.544620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6104.556161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6104.574705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6104.580617] Lustre: Skipped 1 previous similar message [ 6107.753415] Lustre: server umount lustre-MDT0000 complete [ 6110.693453] LustreError: 198317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.721307] LustreError: 198317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6111.584709] LustreError: 197314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788691751 with bad export cookie 1362360323294457984 [ 6111.602951] LustreError: 197314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6111.604900] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6111.933377] Lustre: server umount lustre-MDT0001 complete [ 6122.530623] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 6125.144747] Key type lgssc unregistered [ 6125.455202] LNet: 201183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6125.464700] LNetError: 201183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6125.477841] LNet: Removed LNI 192.168.204.101@tcp [ 6126.386824] Key type .llcrypt unregistered [ 6126.395789] Key type ._llcrypt unregistered [ 6144.024097] Key type ._llcrypt registered [ 6144.026240] Key type .llcrypt registered [ 6144.104844] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 6144.954216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6145.064467] alg: No test for adler32 (adler32-zlib) [ 6146.226341] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 6146.641270] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 6148.343160] Key type lgssc registered [ 6149.869236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6156.932634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6156.960320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6158.319073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6162.283907] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6168.546744] LustreError: 202446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.584347] LustreError: 202446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6172.997711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.388567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6178.572607] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6185.633679] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6191.399335] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6197.996315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6198.367862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6203.806561] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6204.409237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6211.078511] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6218.510559] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6231.962418] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6241.987578] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6251.111699] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6264.801929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6264.816418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6264.852817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6265.835910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.849302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6265.866429] Lustre: Skipped 1 previous similar message [ 6268.265973] Lustre: server umount lustre-OST0000 complete [ 6276.066325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6276.066525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6276.067115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6276.108727] Lustre: Skipped 1 previous similar message [ 6281.184964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6281.194745] Lustre: Skipped 2 previous similar messages [ 6286.303191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6286.309044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6286.322589] Lustre: Skipped 1 previous similar message [ 6286.503238] Lustre: server umount lustre-MDT0000 complete [ 6290.019806] LustreError: 205124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788691929 with bad export cookie 10879036196815422231 [ 6290.020894] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6290.027832] LustreError: 205124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6290.396335] Lustre: server umount lustre-MDT0001 complete [ 6301.706478] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 6304.830968] Key type lgssc unregistered [ 6305.291245] LNet: 207567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6305.307286] LNetError: 207567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6305.335876] LNet: Removed LNI 192.168.204.101@tcp [ 6306.405872] Key type .llcrypt unregistered [ 6306.417132] Key type ._llcrypt unregistered [ 6332.126659] Key type ._llcrypt registered [ 6332.131271] Key type .llcrypt registered [ 6332.236818] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 6333.600557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6333.618636] alg: No test for adler32 (adler32-zlib) [ 6334.679097] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 6334.978051] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 6336.751195] Key type lgssc registered [ 6338.027363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6346.139382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6346.181599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.667076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6353.509791] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6358.006779] LustreError: 208977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6358.040947] LustreError: 208977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6363.114893] LustreError: 208978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6364.377340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6364.711098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6369.638939] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6378.021781] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6385.229407] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6393.356336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6394.150650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6400.765893] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6409.077464] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6417.672826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6436.321093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6436.329582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.361449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6438.370351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6438.378769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6438.395174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6441.426588] Lustre: server umount lustre-OST0000 complete [ 6446.561247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6446.567879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.589492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6446.597827] Lustre: Skipped 1 previous similar message [ 6448.610699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6448.611386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6451.638646] Lustre: server umount lustre-MDT0000 complete [ 6453.728464] LustreError: 208977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6453.739570] LustreError: 208977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6456.131782] LustreError: 208957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788692095 with bad export cookie 14173778632502651408 [ 6456.142287] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6456.164882] LustreError: 208957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6457.091087] Lustre: server umount lustre-MDT0001 complete [ 6470.812532] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 6474.325226] Key type lgssc unregistered [ 6474.648207] LNet: 212690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6474.663958] LNetError: 212690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6475.687553] LNet: Removed LNI 192.168.204.101@tcp [ 6476.875975] Key type .llcrypt unregistered [ 6476.877986] Key type ._llcrypt unregistered [ 6494.618349] Key type ._llcrypt registered [ 6494.621215] Key type .llcrypt registered [ 6494.747380] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 6495.752694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6495.794733] alg: No test for adler32 (adler32-zlib) [ 6496.830810] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 6497.029595] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 6498.679877] Key type lgssc registered [ 6500.062266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6507.558859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6507.573788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6509.247893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6514.153970] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6518.753415] LustreError: 213954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6518.761259] LustreError: 213954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6521.284543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6521.660295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6525.747802] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6534.982462] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6541.597450] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6548.678889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6549.258968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6553.346305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6555.854892] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6563.328431] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6579.179037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6579.196083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.222441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6582.241331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6582.245580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6582.249983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6582.272496] Lustre: Skipped 1 previous similar message [ 6584.368111] Lustre: server umount lustre-OST0000 complete [ 6588.384985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6588.395850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.416740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6592.481178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6592.481334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6594.475176] Lustre: server umount lustre-MDT0000 complete [ 6597.600431] LustreError: 213954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6597.619958] LustreError: 213954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6597.648106] LustreError: 213933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788692237 with bad export cookie 3421571911137234758 [ 6597.650960] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6597.669892] LustreError: 213933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6597.934955] Lustre: server umount lustre-MDT0001 complete [ 6608.220153] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 6611.936871] Key type lgssc unregistered [ 6612.168077] LNet: 217632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6612.185377] LNetError: 217632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6612.206378] LNet: Removed LNI 192.168.204.101@tcp [ 6613.128924] Key type .llcrypt unregistered [ 6613.130915] Key type ._llcrypt unregistered [ 6633.625271] Key type ._llcrypt registered [ 6633.630539] Key type .llcrypt registered [ 6633.839990] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 6634.734851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6634.875975] alg: No test for adler32 (adler32-zlib) [ 6636.056786] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 6636.347230] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 6638.031232] Key type lgssc registered [ 6639.075226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6646.150042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6646.181224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.518437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.211883] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6658.018411] LustreError: 218891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6658.030346] LustreError: 218891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6658.547451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.900390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6663.596645] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6671.097161] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6677.272874] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6684.304261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6684.795844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6692.018384] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6692.993465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6700.632853] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6709.300430] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6709.413617] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6709.689646] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6716.357558] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6723.979788] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6725.597445] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6737.572905] Lustre: Permanently deactivating lustre-OST0001 [ 6737.609924] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6747.214586] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6756.711954] Lustre: Permanently reactivating lustre-OST0001 [ 6756.745289] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6756.751445] Lustre: Skipped 2 previous similar messages [ 6762.525031] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6762.552764] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6762.571833] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6762.601448] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6762.633161] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6772.183894] Lustre: server umount lustre-OST0001 complete [ 6772.724549] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6772.738311] Lustre: Skipped 2 previous similar messages [ 6772.742042] LustreError: 220714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6772.755838] LustreError: 220714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6777.831824] LustreError: 220714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6777.832864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.853555] LustreError: 220714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6777.880776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6777.886453] Lustre: Skipped 1 previous similar message [ 6782.956633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6782.959839] LustreError: 220712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6782.964884] Lustre: Skipped 1 previous similar message [ 6782.992465] LustreError: 220712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6783.462656] Lustre: server umount lustre-OST0000 complete [ 6788.065217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.075595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6788.081309] Lustre: Skipped 2 previous similar messages [ 6788.089850] Lustre: Skipped 1 previous similar message [ 6793.189586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6793.196297] Lustre: Skipped 1 previous similar message [ 6795.781869] Lustre: server umount lustre-MDT0000 complete [ 6798.318643] LustreError: 218892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6798.362719] LustreError: 218892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6800.113949] LustreError: 218873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788692439 with bad export cookie 11217981364069460542 [ 6800.117572] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6800.122795] LustreError: 218873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6800.612107] Lustre: server umount lustre-MDT0001 complete [ 6811.674459] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 6814.268936] Key type lgssc unregistered [ 6814.743493] LNet: 224008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6814.750216] LNetError: 224008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6814.781219] LNet: Removed LNI 192.168.204.101@tcp [ 6815.838941] Key type .llcrypt unregistered [ 6815.840924] Key type ._llcrypt unregistered [ 6858.323418] Key type ._llcrypt registered [ 6858.325578] Key type .llcrypt registered [ 6858.408923] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 6859.405968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6859.574566] alg: No test for adler32 (adler32-zlib) [ 6860.558950] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 6860.748769] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 6862.407564] Key type lgssc registered [ 6863.792257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6872.096175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6872.127642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6873.420947] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6873.459715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6873.714147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6878.582275] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6887.419579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6887.579826] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6887.620189] Lustre: 226219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6887.913736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6892.144661] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6899.408551] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6905.200889] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6911.844183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6912.114865] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6912.371663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6918.563211] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6919.405450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6927.555697] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6931.343603] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6935.805153] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6942.578268] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6951.430823] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6961.487346] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6977.953905] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6977.959581] Lustre: Skipped 1 previous similar message [ 7011.917876] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7011.923230] Lustre: Skipped 3 previous similar messages [ 7081.306534] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7081.315573] Lustre: Skipped 8 previous similar messages [ 7109.107187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7109.111866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7109.118158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7109.132511] Lustre: Skipped 1 previous similar message [ 7113.929084] Lustre: server umount lustre-OST0000 complete [ 7118.305472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7118.313111] LustreError: Skipped 1 previous similar message [ 7118.319210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7118.333853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7118.345411] Lustre: Skipped 1 previous similar message [ 7119.328906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7119.356224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7119.362546] Lustre: Skipped 1 previous similar message [ 7124.788177] Lustre: server umount lustre-MDT0000 complete [ 7128.581338] LustreError: 228081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788692768 with bad export cookie 1276184831100639177 [ 7128.587958] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7128.599460] LustreError: 228081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7129.275187] Lustre: server umount lustre-MDT0001 complete [ 7139.614344] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 7142.433177] Key type lgssc unregistered [ 7142.690911] LNet: 231035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7142.697166] LNetError: 231035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7142.709693] LNet: Removed LNI 192.168.204.101@tcp [ 7143.596200] Key type .llcrypt unregistered [ 7143.598158] Key type ._llcrypt unregistered [ 7166.173674] Key type ._llcrypt registered [ 7166.175741] Key type .llcrypt registered [ 7166.285240] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 7167.497449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7167.650145] alg: No test for adler32 (adler32-zlib) [ 7168.817656] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 7168.984554] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 7170.623311] Key type lgssc registered [ 7172.177850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7179.925715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7179.941714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7181.580341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7186.377684] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7192.034043] LustreError: 232444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7192.069809] LustreError: 232444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7193.882419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7194.422653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7198.399314] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7205.483050] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7211.217858] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7217.705817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7218.027484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7223.839414] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7227.282493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7232.063530] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7250.914902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7250.935544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.965868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7252.965710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7252.974141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7252.991868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7258.103400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7258.117339] Lustre: Skipped 1 previous similar message [ 7263.204466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7263.222174] Lustre: Skipped 2 previous similar messages [ 7264.226496] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7264.345633] Lustre: server umount lustre-OST0000 complete [ 7268.337393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7268.359530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.405671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7271.419559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.654357] Lustre: server umount lustre-MDT0000 complete [ 7278.564787] LustreError: 233509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7278.604217] LustreError: 233509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7278.976117] LustreError: 232426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788692918 with bad export cookie 9103221285964041640 [ 7278.994920] LustreError: 232426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7278.999926] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7279.406242] Lustre: server umount lustre-MDT0001 complete [ 7291.888782] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 7296.300179] Key type lgssc unregistered [ 7296.723807] LNet: 236133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7296.733492] LNetError: 236133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7296.755621] LNet: Removed LNI 192.168.204.101@tcp [ 7297.835074] Key type .llcrypt unregistered [ 7297.843118] Key type ._llcrypt unregistered [ 7323.638978] Key type ._llcrypt registered [ 7323.641485] Key type .llcrypt registered [ 7323.785967] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 7325.074523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7325.217209] alg: No test for adler32 (adler32-zlib) [ 7326.312654] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 7326.634699] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 7328.455184] Key type lgssc registered [ 7329.562310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7598.837704] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 7601.764608] Key type lgssc unregistered [ 7602.090387] LNet: 238154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7602.100137] LNetError: 238154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7602.112809] LNet: Removed LNI 192.168.204.101@tcp [ 7603.118949] Key type .llcrypt unregistered [ 7603.126677] Key type ._llcrypt unregistered [ 7629.882484] Key type ._llcrypt registered [ 7629.884523] Key type .llcrypt registered [ 7629.999288] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 7646.320743] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 7647.929118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7647.954114] alg: No test for adler32 (adler32-zlib) [ 7649.140382] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 7649.427259] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 7651.176162] Key type lgssc registered [ 7653.153705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7691.240311] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7359 sec ========== 07:15:28 (1788693328) [ 7693.354657] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:15:31 (1788693331) === [ 7697.339367] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:15:35 (1788693335) === [ 7740.255295] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 7743.382448] Key type lgssc unregistered [ 7743.782670] LNet: 243390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7743.786993] LNetError: 243390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7743.806462] LNet: Removed LNI 192.168.204.101@tcp [ 7744.740262] Key type .llcrypt unregistered [ 7744.742224] Key type ._llcrypt unregistered