[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 556344813 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002345] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004037] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.010107] LSM: Security Framework initializing [ 0.011063] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.014030] *** VALIDATE selinux *** [ 0.022399] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026780] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029077] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] *** VALIDATE tmpfs *** [ 0.031459] *** VALIDATE proc *** [ 0.032261] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034246] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037025] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041352] debug: unmapping init [mem 0xffffffff87059000-0xffffffff87060fff] [ 0.043161] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044662] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045020] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047010] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052289] rcu: Hierarchical SRCU implementation. [ 0.054474] smp: Bringing up secondary CPUs ... [ 0.055502] x86: Booting SMP configuration: [ 0.056021] .... node #0, CPUs: #1 #2 #3 [ 0.060013] smp: Brought up 1 node, 4 CPUs [ 0.062020] smpboot: Max logical packages: 1 [ 0.063013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.224895] node 0 deferred pages initialised in 157ms [ 0.226213] devtmpfs: initialized [ 0.227235] x86/mm: Memory block size: 128MB [ 0.229828] gcov: version magic: 0x41383552 [ 0.231308] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.232079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.233286] pinctrl core: initialized pinctrl subsystem [ 0.234194] [ 0.234718] ************************************************************* [ 0.235015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.236015] ** ** [ 0.237013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.238015] ** ** [ 0.239015] ** This means that this kernel is built to expose internal ** [ 0.240015] ** IOMMU data structures, which may compromise security on ** [ 0.241013] ** your system. ** [ 0.242015] ** ** [ 0.243014] ** If you see this message and you are not debugging the ** [ 0.244016] ** kernel, report this immediately to your vendor! ** [ 0.245014] ** ** [ 0.246018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247012] ************************************************************* [ 0.248653] NET: Registered protocol family 16 [ 0.249468] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.250063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.251065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.252442] cpuidle: using governor menu [ 0.254821] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.257679] PCI: Using configuration type 1 for base access [ 0.260168] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.274152] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.276048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.281062] cryptd: max_cpu_qlen set to 1000 [ 0.283363] ACPI: Added _OSI(Module Device) [ 0.285017] ACPI: Added _OSI(Processor Device) [ 0.287017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.289016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.295620] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.304338] ACPI: Interpreter enabled [ 0.306081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.308023] ACPI: Using IOAPIC for interrupt routing [ 0.311185] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.315495] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.329400] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.332048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.335021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.339153] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.345709] acpiphp: Slot [2] registered [ 0.347172] acpiphp: Slot [5] registered [ 0.349170] acpiphp: Slot [6] registered [ 0.351139] acpiphp: Slot [7] registered [ 0.353134] acpiphp: Slot [8] registered [ 0.355122] acpiphp: Slot [9] registered [ 0.357158] acpiphp: Slot [10] registered [ 0.359168] acpiphp: Slot [3] registered [ 0.360207] acpiphp: Slot [4] registered [ 0.362107] acpiphp: Slot [11] registered [ 0.364165] acpiphp: Slot [12] registered [ 0.365121] acpiphp: Slot [13] registered [ 0.367109] acpiphp: Slot [14] registered [ 0.368185] acpiphp: Slot [15] registered [ 0.369121] acpiphp: Slot [16] registered [ 0.370000] acpiphp: Slot [17] registered [ 0.370000] acpiphp: Slot [18] registered [ 0.372121] acpiphp: Slot [19] registered [ 0.374112] acpiphp: Slot [20] registered [ 0.376206] acpiphp: Slot [21] registered [ 0.378133] acpiphp: Slot [22] registered [ 0.380103] acpiphp: Slot [23] registered [ 0.382114] acpiphp: Slot [24] registered [ 0.384111] acpiphp: Slot [25] registered [ 0.385099] acpiphp: Slot [26] registered [ 0.387130] acpiphp: Slot [27] registered [ 0.388095] acpiphp: Slot [28] registered [ 0.389105] acpiphp: Slot [29] registered [ 0.391073] acpiphp: Slot [30] registered [ 0.392080] acpiphp: Slot [31] registered [ 0.394075] PCI host bridge to bus 0000:00 [ 0.396021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.398081] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.401022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.404021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.406020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.409029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.411189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.413961] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.417318] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.427937] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.433057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.436023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.439020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.441027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.445606] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.446000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.449052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.452845] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.458017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.473018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.479015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.483164] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.491016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.502015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.520019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.531212] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.537016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.543014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.560021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.568000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.578025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.585000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.605016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.618608] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.632014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.648017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.670016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.682820] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.695015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.706015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.727017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.738299] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.745019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.751016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.768021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.783579] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.787661] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.790500] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.792451] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.794189] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.799086] iommu: Default domain type: Passthrough [ 0.801609] SCSI subsystem initialized [ 0.804188] ACPI: bus type USB registered [ 0.805164] usbcore: registered new interface driver usbfs [ 0.808145] usbcore: registered new interface driver hub [ 0.810117] usbcore: registered new device driver usb [ 0.812241] pps_core: LinuxPPS API ver. 1 registered [ 0.814017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.818088] PTP clock support registered [ 0.821076] EDAC MC: Ver: 3.0.0 [ 0.824162] PCI: Using ACPI for IRQ routing [ 0.825827] NetLabel: Initializing [ 0.827015] NetLabel: domain hash size = 128 [ 0.828009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.829088] NetLabel: unlabeled traffic allowed by default [ 0.831148] vgaarb: loaded [ 0.832411] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.835070] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.841000] clocksource: Switched to clocksource kvm-clock [ 0.966671] VFS: Disk quotas dquot_6.6.0 [ 0.968423] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.971571] *** VALIDATE ramfs *** [ 0.973139] *** VALIDATE hugetlbfs *** [ 0.975225] pnp: PnP ACPI init [ 0.978262] pnp: PnP ACPI: found 6 devices [ 1.002158] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.007078] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.008986] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.010723] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.012695] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.015007] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.018580] NET: Registered protocol family 2 [ 1.022357] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.029784] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.033242] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.038619] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.042651] TCP: Hash tables configured (established 65536 bind 65536) [ 1.045477] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.048612] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.051813] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.054748] NET: Registered protocol family 1 [ 1.057413] RPC: Registered named UNIX socket transport module. [ 1.059657] RPC: Registered udp transport module. [ 1.061610] RPC: Registered tcp transport module. [ 1.063267] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.065469] NET: Registered protocol family 44 [ 1.067284] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.069478] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.071340] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.073420] PCI: CLS 0 bytes, default 64 [ 1.075628] Unpacking initramfs... [ 2.533284] debug: unmapping init [mem 0xffff9b737cc54000-0xffff9b737ffbffff] [ 2.537407] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.540013] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.543087] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.052204] Initialise system trusted keyrings [ 3.054447] Key type blacklist registered [ 3.056623] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.066563] zbud: loaded [ 3.070385] *** VALIDATE nfs *** [ 3.071920] *** VALIDATE nfs4 *** [ 3.073519] pstore: using deflate compression [ 3.077505] Platform Keyring initialized [ 3.196938] NET: Registered protocol family 38 [ 3.199152] Key type asymmetric registered [ 3.201078] Asymmetric key parser 'x509' registered [ 3.203566] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.207605] io scheduler mq-deadline registered [ 3.209614] io scheduler kyber registered [ 3.211617] io scheduler bfq registered [ 3.214588] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.217961] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.221101] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.223931] ACPI: Power Button [PWRF] [ 3.229281] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.237441] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.253764] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.261516] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.275145] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.303153] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.331157] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.337104] Non-volatile memory driver v1.3 [ 3.339083] Linux agpgart interface v0.103 [ 3.377241] virtio_blk virtio1: [vda] 150016 512-byte logical blocks (76.8 MB/73.3 MiB) [ 3.380896] vda: detected capacity change from 0 to 76808192 [ 3.396092] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.399164] vdb: detected capacity change from 0 to 1073741824 [ 3.413469] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.416616] vdc: detected capacity change from 0 to 2621440000 [ 3.435600] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.440292] vdd: detected capacity change from 0 to 2621440000 [ 3.461606] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.465314] vde: detected capacity change from 0 to 4294967296 [ 3.482182] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.486155] vdf: detected capacity change from 0 to 4294967296 [ 3.496442] libphy: Fixed MDIO Bus: probed [ 3.503501] usbcore: registered new interface driver usbserial_generic [ 3.506687] usbserial: USB Serial support registered for generic [ 3.509670] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.515328] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.517593] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.521900] mousedev: PS/2 mouse device common for all mice [ 3.525192] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.529616] rtc_cmos 00:05: RTC can wake from S4 [ 3.533738] rtc_cmos 00:05: registered as rtc0 [ 3.534147] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.537923] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.544295] intel_pstate: CPU model not supported [ 3.545895] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.550494] hid: raw HID events driver (C) Jiri Kosina [ 3.553288] usbcore: registered new interface driver usbhid [ 3.556015] usbhid: USB HID core driver [ 3.557703] drop_monitor: Initializing network drop monitor service [ 3.559826] Initializing XFRM netlink socket [ 3.562039] NET: Registered protocol family 10 [ 3.565330] Segment Routing with IPv6 [ 3.567086] NET: Registered protocol family 17 [ 3.569333] mpls_gso: MPLS GSO support [ 3.575330] RAS: Correctable Errors collector initialized. [ 3.578160] AVX version of gcm_enc/dec engaged. [ 3.580267] AES CTR mode by8 optimization enabled [ 3.665110] sched_clock: Marking stable (3665036177, 0)->(4604244390, -939208213) [ 3.669302] registered taskstats version 1 [ 3.671594] Loading compiled-in X.509 certificates [ 3.674398] zswap: loaded using pool lzo/zbud [ 3.704152] Key type big_key registered [ 3.716858] Key type encrypted registered [ 3.718758] ima: No TPM chip found, activating TPM-bypass! [ 3.721113] ima: Allocated hash algorithm: sha1 [ 3.722665] ima: No architecture policies found [ 3.724575] evm: Initialising EVM extended attributes: [ 3.726598] evm: security.selinux [ 3.728106] evm: security.ima [ 3.729318] evm: security.capability [ 3.730519] evm: HMAC attrs: 0x1 [ 3.732847] rtc_cmos 00:05: setting system clock to 2026-09-12 03:56:22 UTC (1789185382) [ 3.739996] debug: unmapping init [mem 0xffffffff88003000-0xffffffff881fffff] [ 3.742781] debug: unmapping init [mem 0xffffffff86d82000-0xffffffff87058fff] [ 3.750137] Write protecting the kernel read-only data: 28672k [ 3.755297] debug: unmapping init [mem 0xffffffff85403000-0xffffffff855fffff] [ 3.759080] debug: unmapping init [mem 0xffffffff85d14000-0xffffffff85dfffff] [ 3.802175] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.812454] systemd[1]: Detected virtualization kvm. [ 3.814944] systemd[1]: Detected architecture x86-64. [ 3.817171] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.849416] systemd[1]: No hostname configured. [ 3.851200] systemd[1]: Set hostname to . [ 3.853251] random: systemd: uninitialized urandom read (16 bytes read) [ 3.855950] systemd[1]: Initializing machine ID from random generator. [ 3.901943] random: ln: uninitialized urandom read (6 bytes read) [ 3.993702] random: systemd: uninitialized urandom read (16 bytes read) [ 3.996416] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.001749] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.006712] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.733318] device-mapper: uevent: version 1.0.3 [ 4.737673] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.496914] random: fast init done [ 5.504231] virtio_net virtio0 ens2: renamed from eth0 [ 5.560722] scsi host0: ata_piix [ 5.598434] scsi host1: ata_piix [ 5.600240] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.602669] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.090202] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.235797] random: crng init done [ 10.237356] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.852609] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.069570] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.368360] SELinux: Disabled at runtime. [ 12.430888] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.440345] systemd[1]: Detected virtualization kvm. [ 12.442526] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.013487] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.017556] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.023478] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.029305] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.033817] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.042378] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.050355] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 13.109991] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.533882] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.855305] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.890543] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.001535] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.014188] EDAC sbridge: Ver: 1.1.2 [ 16.023531] Key type dns_resolver registered [ 16.329530] NFS: Registering the id_resolver key type [ 16.331901] Key type id_resolver registered [ 16.333622] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg454-server login: [ 65.568853] libcfs: loading out-of-tree module taints kernel. [ 65.606718] Key type ._llcrypt registered [ 65.609089] Key type .llcrypt registered [ 65.733956] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 84.273583] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 86.169473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.180872] alg: No test for adler32 (adler32-zlib) [ 87.701751] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 88.371654] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 90.087315] Key type lgssc registered [ 91.374738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 112.048949] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 158.703111] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 164.338009] hrtimer: interrupt took 5139491 ns [ 173.559169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 173.641306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.868719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 174.896181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 175.007748] Lustre: lustre-MDT0000: new disk, initializing [ 175.111988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 175.144645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 180.070761] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.345082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.474698] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 193.533459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 193.544132] Lustre: Skipped 1 previous similar message [ 193.686448] Lustre: lustre-MDT0001: new disk, initializing [ 193.809137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 193.846983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 193.861870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 198.384897] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.550055] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 215.078499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.393898] Lustre: lustre-OST0000: new disk, initializing [ 215.396992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.405055] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 215.413538] Lustre: 8456:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.486992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.715547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.725455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.756636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 223.058615] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.567945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.732588] Lustre: lustre-OST0001: new disk, initializing [ 238.738009] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 238.746992] Lustre: 9529:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.821073] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.247876] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.897317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 247.921749] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 248.048635] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 258.520891] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 264.746284] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 271.853402] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing check_logdir /tmp/testlogs/ [ 277.255918] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing yml_node [ 283.178174] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 287.370766] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 291.171914] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 293.631852] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 12 00:01:09 EDT 2026 [ 314.197436] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 315.954754] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 335.328724] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.335652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.346420] Lustre: Skipped 3 previous similar messages [ 335.356988] Lustre: Skipped 3 previous similar messages [ 338.844622] Lustre: server umount lustre-MDT0000 complete [ 343.002782] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185721 with bad export cookie 13164609056380062725 [ 343.009351] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.015752] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 345.573480] LustreError: 6525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.573969] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.589946] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 345.601968] LustreError: 6525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 345.634080] Lustre: Skipped 1 previous similar message [ 348.587645] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.593463] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.623053] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 348.655386] Lustre: Skipped 2 previous similar messages [ 352.673351] LustreError: 12359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.676107] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.693155] LustreError: 12359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 352.709615] Lustre: Skipped 1 previous similar message [ 357.343876] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 357.581343] Lustre: server umount lustre-MDT0001 complete [ 362.147717] Lustre: server umount lustre-OST0000 complete [ 366.870996] Lustre: server umount lustre-OST0001 complete [ 374.166946] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 384.866712] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 431.368384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.612244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 431.641942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 431.718746] Lustre: lustre-MDT0000: new disk, initializing [ 431.827403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.859067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 437.470063] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 450.692362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.854840] Lustre: 15100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 450.985864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 450.993496] Lustre: Skipped 1 previous similar message [ 451.147678] Lustre: lustre-MDT0001: new disk, initializing [ 451.273169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 451.320373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 451.346564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 456.689900] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 467.936752] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 475.286395] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 483.290512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 483.801674] Lustre: lustre-OST0000: new disk, initializing [ 483.815555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 483.823672] Lustre: 17071:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 483.964245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.081527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 485.098545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 485.256134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 490.894955] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 502.577389] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 507.688825] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 507.925110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 512.781930] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 513.064912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 516.064291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 516.068595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.092800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 516.106647] Lustre: Skipped 1 previous similar message [ 516.123364] Lustre: Skipped 1 previous similar message [ 529.376581] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 529.608554] Lustre: server umount lustre-OST0000 complete [ 536.548873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.555462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 536.578105] Lustre: Skipped 1 previous similar message [ 536.594341] Lustre: Skipped 5 previous similar messages [ 540.131877] Lustre: server umount lustre-MDT0000 complete [ 541.667200] LustreError: 15112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.702756] LustreError: 15112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 544.065049] LustreError: 15093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185922 with bad export cookie 13164609056380064258 [ 544.071873] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.081367] LustreError: 15093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 544.644757] Lustre: server umount lustre-MDT0001 complete [ 548.791080] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 00:05:24 (1789185924) [ 556.733589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.510761] LustreError: 19338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.616962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 562.339785] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 567.778581] LustreError: 19339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.799781] LustreError: 19339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 569.714961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.670177] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 580.890801] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 586.166985] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 591.868288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 598.498841] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 607.018096] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 615.905168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 615.912829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.929787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 617.952544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 621.135947] Lustre: server umount lustre-OST0000 complete [ 626.656757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 626.662415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.671711] Lustre: Skipped 1 previous similar message [ 631.420983] Lustre: server umount lustre-MDT0000 complete [ 633.317593] LustreError: 19335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.337600] LustreError: 19335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 635.733606] LustreError: 19320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186014 with bad export cookie 13164609056380065140 [ 635.752582] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 635.762908] LustreError: 19320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 636.154569] Lustre: server umount lustre-MDT0001 complete [ 647.387451] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 649.976438] Key type lgssc unregistered [ 650.223752] LNet: 22869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.230885] LNetError: 22869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 650.249802] LNet: Removed LNI 192.168.204.154@tcp [ 651.143173] Key type .llcrypt unregistered [ 651.148256] Key type ._llcrypt unregistered [ 674.206455] Key type ._llcrypt registered [ 674.208613] Key type .llcrypt registered [ 674.305682] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 674.932220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 674.957945] alg: No test for adler32 (adler32-zlib) [ 675.940633] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 676.195254] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 677.847262] Key type lgssc registered [ 678.823481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 685.234235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 685.248416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.637616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 689.920058] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 696.804057] LustreError: 24282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 696.823317] LustreError: 24282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 697.050666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.394166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 700.777719] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 706.899121] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 711.678430] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 716.951631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 717.262632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 722.019532] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 727.333588] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 731.640275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 741.858201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 741.867459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.872323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 741.886601] Lustre: Skipped 1 previous similar message [ 741.894733] Lustre: Skipped 1 previous similar message [ 744.546565] Lustre: server umount lustre-OST0000 complete [ 748.512208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 748.522631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.532308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 752.097299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 752.103817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 753.711564] Lustre: server umount lustre-MDT0000 complete [ 756.590354] LustreError: 24263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186135 with bad export cookie 10783044318151722063 [ 756.593571] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 756.601993] LustreError: 24263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 756.903969] Lustre: server umount lustre-MDT0001 complete [ 767.016536] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 769.614312] Key type lgssc unregistered [ 769.879866] LNet: 28254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.887706] LNetError: 28254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.908770] LNet: Removed LNI 192.168.204.154@tcp [ 770.724094] Key type .llcrypt unregistered [ 770.727397] Key type ._llcrypt unregistered [ 793.550980] Key type ._llcrypt registered [ 793.553799] Key type .llcrypt registered [ 793.659912] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 794.725960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 794.752656] alg: No test for adler32 (adler32-zlib) [ 795.799265] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 796.058657] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 797.791430] Key type lgssc registered [ 798.722500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 804.795190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 804.814350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 806.116727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 809.753488] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 815.590326] LustreError: 29661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 815.606815] LustreError: 29661:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 817.891664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.226622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 821.649542] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 827.767224] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 833.728755] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 845.029193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 845.372855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.481515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 850.994990] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 856.879336] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 867.812927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 867.828421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 867.830806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 867.836423] Lustre: Skipped 1 previous similar message [ 869.988151] Lustre: server umount lustre-OST0000 complete [ 874.468915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 874.482921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.493994] Lustre: Skipped 1 previous similar message [ 874.502953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.050757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.050908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.350317] Lustre: server umount lustre-MDT0000 complete [ 882.470413] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186261 with bad export cookie 8125531273477088007 [ 882.472364] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 882.482294] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 882.906172] Lustre: server umount lustre-MDT0001 complete [ 890.813833] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 892.968670] Key type lgssc unregistered [ 893.235611] LNet: 33537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.247928] LNetError: 33537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.264590] LNet: Removed LNI 192.168.204.154@tcp [ 893.976603] Key type .llcrypt unregistered [ 893.980888] Key type ._llcrypt unregistered [ 913.002477] Key type ._llcrypt registered [ 913.006248] Key type .llcrypt registered [ 913.131589] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 913.814156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 913.853717] alg: No test for adler32 (adler32-zlib) [ 914.883945] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 915.092304] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 916.727278] Key type lgssc registered [ 917.653815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 924.103592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 924.126255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.507813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.092350] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 935.908111] LustreError: 34927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.921038] LustreError: 34927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 936.746533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.089262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 941.357805] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 947.615403] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 952.601391] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 958.359471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 958.652410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.939339] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 969.579379] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 970.928912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 981.477776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.478072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 981.492754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 981.501069] Lustre: Skipped 1 previous similar message [ 983.773646] Lustre: server umount lustre-OST0000 complete [ 988.133302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 988.145875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.170427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 988.176664] Lustre: Skipped 1 previous similar message [ 991.713500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.714652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.713249] Lustre: server umount lustre-MDT0000 complete [ 996.835477] LustreError: 34928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.852697] LustreError: 34928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 996.876159] LustreError: 34907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186375 with bad export cookie 11033033068505643869 [ 996.881112] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 996.889741] LustreError: 34907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 997.126231] Lustre: server umount lustre-MDT0001 complete [ 1005.816391] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1007.748535] Key type lgssc unregistered [ 1007.955426] LNet: 38458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.966227] LNetError: 38458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.986882] LNet: Removed LNI 192.168.204.154@tcp [ 1008.628287] Key type .llcrypt unregistered [ 1008.633421] Key type ._llcrypt unregistered [ 1027.469856] Key type ._llcrypt registered [ 1027.474458] Key type .llcrypt registered [ 1027.591114] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1028.387344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1028.455368] alg: No test for adler32 (adler32-zlib) [ 1029.496242] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1029.686758] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1031.343247] Key type lgssc registered [ 1032.200435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1038.570963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1038.584052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.892573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.149694] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1049.779036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1050.107190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1053.884878] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1060.195513] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1064.729774] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1070.360686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1070.608788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.859251] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1082.528277] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1085.994405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1093.221558] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1096.161108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1096.163402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.168960] LustreError: Skipped 1 previous similar message [ 1096.173839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1096.177701] Lustre: Skipped 1 previous similar message [ 1096.197905] Lustre: Skipped 1 previous similar message [ 1098.302393] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1101.285981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1101.296785] Lustre: Skipped 1 previous similar message [ 1105.887185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1106.040249] Lustre: server umount lustre-OST0000 complete [ 1124.317182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.325758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1124.337117] Lustre: Skipped 1 previous similar message [ 1124.354814] Lustre: Skipped 2 previous similar messages [ 1130.587887] Lustre: server umount lustre-MDT0000 complete [ 1131.496859] LustreError: 39843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1131.509413] LustreError: 39843:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1133.520497] LustreError: 39829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186512 with bad export cookie 7536179214642275831 [ 1133.526223] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1133.540512] LustreError: 39829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1139.785978] Lustre: server umount lustre-MDT0001 complete [ 1149.698438] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1152.161988] Key type lgssc unregistered [ 1152.494364] LNet: 43561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.509174] LNetError: 43561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.526789] LNet: Removed LNI 192.168.204.154@tcp [ 1153.247411] Key type .llcrypt unregistered [ 1153.250499] Key type ._llcrypt unregistered [ 1171.753125] Key type ._llcrypt registered [ 1171.756283] Key type .llcrypt registered [ 1171.864028] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1172.412409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1172.424757] alg: No test for adler32 (adler32-zlib) [ 1173.429957] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1173.693179] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1175.362637] Key type lgssc registered [ 1176.211748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1181.946630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1181.975556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1183.374242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.940962] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1193.955515] LustreError: 44949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.972393] LustreError: 44949:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1194.014778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.346783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1197.648327] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1204.222822] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1208.660466] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1214.016803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1214.298829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.588641] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1225.863352] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1228.664074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1237.599355] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1238.396940] Lustre: server umount lustre-MDT0000 complete [ 1239.013153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.014455] LustreError: 46237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.024730] Lustre: Skipped 2 previous similar messages [ 1239.052792] LustreError: 46237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1241.214947] LustreError: 44929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186619 with bad export cookie 4917302219681421322 [ 1241.221056] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.225016] LustreError: 44929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1242.690993] LustreError: 47438:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.719511] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 1244.129668] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.139845] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1246.178463] LustreError: 44945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.181156] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1246.192776] LustreError: 44945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1247.510753] Lustre: server umount lustre-MDT0001 complete [ 1267.642764] Lustre: server umount lustre-OST0000 complete [ 1279.082991] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1280.965738] Key type lgssc unregistered [ 1281.167903] LNet: 48807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1281.174654] LNetError: 48807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1281.192828] LNet: Removed LNI 192.168.204.154@tcp [ 1281.792381] Key type .llcrypt unregistered [ 1281.796415] Key type ._llcrypt unregistered [ 1299.817986] Key type ._llcrypt registered [ 1299.821177] Key type .llcrypt registered [ 1299.906977] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1300.624739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1300.639936] alg: No test for adler32 (adler32-zlib) [ 1301.601706] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1301.771981] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1303.399607] Key type lgssc registered [ 1304.268941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1309.901633] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1309.920956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1325.407417] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1330.527705] LustreError: 50169:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 1357.215934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1362.415330] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1367.992772] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1532.580472] Lustre: server umount lustre-OST0000 complete [ 1541.773161] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1543.473230] Key type lgssc unregistered [ 1543.676628] LNet: 51842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.682538] LNetError: 51842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.697668] LNet: Removed LNI 192.168.204.154@tcp [ 1544.267087] Key type .llcrypt unregistered [ 1544.269560] Key type ._llcrypt unregistered [ 1558.554400] Key type ._llcrypt registered [ 1558.557840] Key type .llcrypt registered [ 1558.633462] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1559.235942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1559.294656] alg: No test for adler32 (adler32-zlib) [ 1560.230085] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1560.384165] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1562.007210] Key type lgssc registered [ 1562.664407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1567.987877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1568.004246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.319849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.106753] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1578.612320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1578.873762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1581.693904] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1586.653861] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1590.202491] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1594.581610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1594.800513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.821720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1598.759114] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1604.084860] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1606.112718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1606.121176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1606.136191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1609.191902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.193159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1609.205548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1609.212814] Lustre: Skipped 1 previous similar message [ 1611.910969] Lustre: server umount lustre-OST0000 complete [ 1614.816085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1614.821499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1614.829520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1619.425572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1619.427146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1620.689881] Lustre: server umount lustre-MDT0000 complete [ 1623.032841] LustreError: 55072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187001 with bad export cookie 2888324720770763241 [ 1623.034388] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1623.038380] LustreError: 55072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1623.224327] Lustre: server umount lustre-MDT0001 complete [ 1629.814301] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1631.600164] Key type lgssc unregistered [ 1631.813308] LNet: 56759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.824416] LNetError: 56759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.838607] LNet: Removed LNI 192.168.204.154@tcp [ 1632.437649] Key type .llcrypt unregistered [ 1632.441937] Key type ._llcrypt unregistered [ 1647.019873] Key type ._llcrypt registered [ 1647.024261] Key type .llcrypt registered [ 1647.122826] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1647.676161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1647.691639] alg: No test for adler32 (adler32-zlib) [ 1648.661146] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1648.831625] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1650.463192] Key type lgssc registered [ 1651.252264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1656.002763] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1656.021033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1671.519497] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1676.639364] LustreError: 58121:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 1703.263303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.587305] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1710.629380] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1714.594864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1714.921423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.966861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1717.195365] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1722.273475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.534463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1725.498279] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1730.417508] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1733.874225] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1735.651972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1735.656665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.665326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1738.210407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1738.214289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.215078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1741.300869] Lustre: server umount lustre-OST0000 complete [ 1758.690849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.692393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1758.697899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.705190] Lustre: Skipped 2 previous similar messages [ 1764.217202] Lustre: server umount lustre-MDT0000 complete [ 1764.832245] LustreError: 59161:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1764.851711] LustreError: 59161:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 1766.758493] LustreError: 58129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187145 with bad export cookie 15758452190645206135 [ 1766.763903] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1766.767812] LustreError: 58129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1781.215138] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1781.384422] Lustre: server umount lustre-MDT0001 complete [ 1787.559402] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1789.267903] Key type lgssc unregistered [ 1789.436558] LNet: 61862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.444199] LNetError: 61862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.457476] LNet: Removed LNI 192.168.204.154@tcp [ 1790.034468] Key type .llcrypt unregistered [ 1790.036448] Key type ._llcrypt unregistered [ 1804.051031] Key type ._llcrypt registered [ 1804.053142] Key type .llcrypt registered [ 1804.112931] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1804.642046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1804.705353] alg: No test for adler32 (adler32-zlib) [ 1805.633835] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1805.804675] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1807.431196] Key type lgssc registered [ 1808.076477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1812.989089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1813.011259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.260880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.542447] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1821.473870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.716766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1824.099120] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1828.403425] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1831.794772] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1835.411977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1835.623743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1839.303995] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1839.719753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1843.731165] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1860.576305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1860.581132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.588735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1862.626530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1862.631798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1862.638806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.712390] Lustre: server umount lustre-OST0000 complete [ 1867.745722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1867.746927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.748598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1867.748605] Lustre: Skipped 1 previous similar message [ 1867.772244] Lustre: Skipped 1 previous similar message [ 1872.864915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.870584] Lustre: Skipped 2 previous similar messages [ 1873.286052] Lustre: server umount lustre-MDT0000 complete [ 1875.491520] LustreError: 63229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187254 with bad export cookie 8764531916265896196 [ 1875.494544] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1875.502674] LustreError: 63229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1875.671307] Lustre: server umount lustre-MDT0001 complete [ 1882.212459] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1883.862178] Key type lgssc unregistered [ 1884.019706] LNet: 66797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.024718] LNetError: 66797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.033430] LNet: Removed LNI 192.168.204.154@tcp [ 1884.575833] Key type .llcrypt unregistered [ 1884.580664] Key type ._llcrypt unregistered [ 1899.384901] Key type ._llcrypt registered [ 1899.386589] Key type .llcrypt registered [ 1899.430449] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1899.940835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1899.948297] alg: No test for adler32 (adler32-zlib) [ 1900.906603] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1901.093233] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1902.744416] Key type lgssc registered [ 1903.444603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1909.086602] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 00:28:06 (1789187286) [ 1913.155743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1913.172781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.433224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.742853] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1921.927500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.156264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1924.288383] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1928.462954] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1931.706985] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1935.933290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1936.123715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.643418] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1940.202495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1943.736389] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1951.372956] Lustre: Failing over lustre-MDT0000 [ 1951.578162] Lustre: server umount lustre-MDT0000 complete [ 1951.715328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.720117] LustreError: 69933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.727503] Lustre: Skipped 2 previous similar messages [ 1951.743164] LustreError: 69933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1955.405433] LustreError: 69933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1956.834583] LustreError: 68487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1956.845866] LustreError: 68487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1957.191091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.263188] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1957.346799] Lustre: *** cfs_fail_loc=135, val=0*** [ 1957.393632] LustreError: 71577:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1957.397276] LustreError: 71577:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -2 [ 1957.402538] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1957.402538] [ 1957.407611] LustreError: MGC192.168.204.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1957.416866] LustreError: 71565:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1957.423934] LustreError: 71565:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 1957.429717] LustreError: 71565:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 1957.433636] LustreError: 71565:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1957.448172] Lustre: server umount lustre-MDT0000 complete [ 1957.451571] LustreError: 71565:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1960.507595] LustreError: 69556:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1960.896965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.972488] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1961.129272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.168103] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1963.364594] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1965.629199] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1966.567534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1966.580469] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1966.618224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1971.681560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.685489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1971.692313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1971.700280] Lustre: Skipped 1 previous similar message [ 1974.499861] Lustre: server umount lustre-OST0000 complete [ 1976.800491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1976.800848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1976.803663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1976.818756] Lustre: Skipped 2 previous similar messages [ 1981.921678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1981.926350] Lustre: Skipped 2 previous similar messages [ 1982.664545] Lustre: server umount lustre-MDT0000 complete [ 1984.767494] LustreError: 68473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187363 with bad export cookie 17614724186799827928 [ 1984.774771] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1984.776930] LustreError: 68473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1984.947953] Lustre: server umount lustre-MDT0001 complete [ 1990.012138] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1991.395875] Key type lgssc unregistered [ 1991.539750] LNet: 73350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.545451] LNetError: 73350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.557221] LNet: Removed LNI 192.168.204.154@tcp [ 1991.948890] Key type .llcrypt unregistered [ 1991.950819] Key type ._llcrypt unregistered [ 2004.656635] Key type ._llcrypt registered [ 2004.660417] Key type .llcrypt registered [ 2004.718534] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2005.197411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2005.254201] alg: No test for adler32 (adler32-zlib) [ 2006.188835] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 2006.334829] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2007.967259] Key type lgssc registered [ 2008.613160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2013.257445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2013.268142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.523814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.000583] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2021.866290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.089573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2024.225394] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2028.128929] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2030.956164] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2034.416859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2034.586442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2037.379896] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2039.719253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2041.118344] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2048.556075] Lustre: Failing over lustre-MDT0000 [ 2048.721657] Lustre: server umount lustre-MDT0000 complete [ 2052.072027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.081459] Lustre: Skipped 1 previous similar message [ 2052.082793] LustreError: 77227:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.098294] LustreError: 77227:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2052.694287] LustreError: 77227:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.179512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.264349] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2054.359186] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2054.434596] LustreError: 77823:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2054.439645] LustreError: 77823:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -9 [ 2054.448118] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2054.448118] [ 2054.454255] LustreError: MGC192.168.204.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2054.465570] LustreError: 77811:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2054.470499] LustreError: 77811:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2054.474632] LustreError: 77811:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2054.478344] LustreError: 77811:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2054.494707] Lustre: server umount lustre-MDT0000 complete [ 2054.498435] LustreError: 77811:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2057.184763] LustreError: 77132:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2057.194305] LustreError: 77132:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2058.626795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.698850] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2058.880335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.923749] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2061.572629] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2062.906932] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2064.360440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2064.376326] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2064.415540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2067.425969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2067.432885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.439665] Lustre: Skipped 1 previous similar message [ 2067.443347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2069.473817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2069.475334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.475567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2072.447827] Lustre: server umount lustre-OST0000 complete [ 2074.595061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.596300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2074.602951] Lustre: Skipped 1 previous similar message [ 2074.603309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2074.607159] Lustre: Skipped 1 previous similar message [ 2079.712189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2079.715233] Lustre: Skipped 2 previous similar messages [ 2080.744084] Lustre: server umount lustre-MDT0000 complete [ 2082.716461] LustreError: 76577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187461 with bad export cookie 18093364545913456851 [ 2082.719177] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2082.723961] LustreError: 76577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2082.918202] Lustre: server umount lustre-MDT0001 complete [ 2088.280976] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2089.870851] Key type lgssc unregistered [ 2090.041925] LNet: 79599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.048732] LNetError: 79599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.055385] LNet: Removed LNI 192.168.204.154@tcp [ 2090.542826] Key type .llcrypt unregistered [ 2090.545176] Key type ._llcrypt unregistered [ 2096.279201] Key type ._llcrypt registered [ 2096.281050] Key type .llcrypt registered [ 2096.339940] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2096.490278] Key type .llcrypt unregistered [ 2096.494114] Key type ._llcrypt unregistered [ 2109.657930] Key type ._llcrypt registered [ 2109.660651] Key type .llcrypt registered [ 2109.736443] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2110.188072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2110.268494] alg: No test for adler32 (adler32-zlib) [ 2111.219414] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 2111.380200] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2113.007343] Key type lgssc registered [ 2113.670769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2118.591579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2118.604804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2119.926617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.620426] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2127.104058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.336868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2129.501273] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2133.533490] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2136.387635] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2139.817647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2140.012816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.105183] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2145.126838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2146.876327] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2157.538797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.539584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2157.541216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2157.548021] Lustre: Skipped 1 previous similar message [ 2162.659057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2162.664287] Lustre: Skipped 2 previous similar messages [ 2162.999265] Lustre: server umount lustre-OST0000 complete [ 2167.780142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.783096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2167.794962] Lustre: Skipped 1 previous similar message [ 2171.182656] Lustre: server umount lustre-MDT0000 complete [ 2172.897119] LustreError: 81466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.908622] LustreError: 81466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2173.269162] LustreError: 81451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187552 with bad export cookie 16144875714580953476 [ 2173.274300] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2173.278170] LustreError: 81451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2173.442894] Lustre: server umount lustre-MDT0001 complete [ 2178.442498] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2180.017204] Key type lgssc unregistered [ 2180.208578] LNet: 85000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.212964] LNetError: 85000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2180.229593] LNet: Removed LNI 192.168.204.154@tcp [ 2180.662516] Key type .llcrypt unregistered [ 2180.664321] Key type ._llcrypt unregistered [ 2192.098540] Key type ._llcrypt registered [ 2192.100168] Key type .llcrypt registered [ 2192.149514] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2192.672151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2192.738864] alg: No test for adler32 (adler32-zlib) [ 2193.638515] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 2193.766632] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2195.367798] Key type lgssc registered [ 2195.982894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2199.521985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2199.531300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.762806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.570124] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2206.510287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.707596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2208.714345] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2212.464617] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2215.277544] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2218.911552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2219.094155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.050384] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2225.255968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2225.795540] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2235.873806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2235.877377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.887228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2236.896065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2236.898823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2236.902594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.630444] Lustre: server umount lustre-OST0000 complete [ 2242.530888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2242.535290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.543327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.548468] Lustre: Skipped 1 previous similar message [ 2247.137450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.139814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.596311] Lustre: server umount lustre-MDT0000 complete [ 2250.593213] LustreError: 88875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187629 with bad export cookie 9055131713516643885 [ 2250.597809] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2250.600322] LustreError: 88875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2250.747535] Lustre: server umount lustre-MDT0001 complete [ 2256.177341] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2257.557090] Key type lgssc unregistered [ 2257.714342] LNet: 89914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2257.720600] LNetError: 89914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2257.729501] LNet: Removed LNI 192.168.204.154@tcp [ 2258.123259] Key type .llcrypt unregistered [ 2258.125087] Key type ._llcrypt unregistered [ 2270.976560] Key type ._llcrypt registered [ 2270.981621] Key type .llcrypt registered [ 2271.038683] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2271.546612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.579706] alg: No test for adler32 (adler32-zlib) [ 2272.499478] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 2272.633499] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2274.255571] Key type lgssc registered [ 2275.085578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2279.393110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2279.410380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.717693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.957965] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2288.075467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.298189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2290.950241] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2295.515344] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2298.692320] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2302.696956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2302.915566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2306.033314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2306.439481] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2310.724643] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2324.453713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.454791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2324.456499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2324.467609] Lustre: Skipped 1 previous similar message [ 2329.158065] Lustre: server umount lustre-OST0000 complete [ 2334.692586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2334.692624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.695902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.695910] Lustre: Skipped 1 previous similar message [ 2334.712872] Lustre: Skipped 1 previous similar message [ 2337.629994] Lustre: server umount lustre-MDT0000 complete [ 2339.746898] LustreError: 91280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187718 with bad export cookie 17473519712359387615 [ 2339.749355] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2339.754634] LustreError: 91280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2339.946679] Lustre: server umount lustre-MDT0001 complete [ 2346.076144] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2347.658972] Key type lgssc unregistered [ 2347.854685] LNet: 94848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2347.860982] LNetError: 94848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2347.870832] LNet: Removed LNI 192.168.204.154@tcp [ 2348.408467] Key type .llcrypt unregistered [ 2348.410112] Key type ._llcrypt unregistered [ 2361.951394] Key type ._llcrypt registered [ 2361.953568] Key type .llcrypt registered [ 2362.014778] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2362.478084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.532967] alg: No test for adler32 (adler32-zlib) [ 2363.444953] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 2363.578060] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2365.199264] Key type lgssc registered [ 2365.877151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2370.393879] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2370.410988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2385.887523] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2391.007284] LustreError: 96207:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 2417.631215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2421.038518] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2424.676603] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2427.511347] Lustre: server umount lustre-OST0000 complete [ 2431.302158] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 00:36:48 (1789187808) [ 2434.624776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.879448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.977486] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2440.163372] LustreError: 97844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.179778] LustreError: 97844:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2441.476558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.683988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2443.985879] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2447.786339] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2450.756561] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2454.115242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2454.312942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.232465] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2459.434328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2460.936058] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2471.911685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.912943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2471.922928] Lustre: Skipped 1 previous similar message [ 2472.738784] Lustre: server umount lustre-OST0000 complete [ 2476.649139] Lustre: server umount lustre-MDT0000 complete [ 2477.025898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.033219] LustreError: 98571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.039824] LustreError: 98571:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2478.309786] LustreError: 97823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187857 with bad export cookie 13915063506672875031 [ 2478.312479] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2478.316501] LustreError: 97823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2478.459862] Lustre: server umount lustre-MDT0001 complete [ 2483.157311] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2484.420752] Key type lgssc unregistered [ 2484.567647] LNet: 101530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.572607] LNetError: 101530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2484.580525] LNet: Removed LNI 192.168.204.154@tcp [ 2484.977158] Key type .llcrypt unregistered [ 2484.979784] Key type ._llcrypt unregistered [ 2505.936107] Key type ._llcrypt registered [ 2505.937849] Key type .llcrypt registered [ 2506.001315] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2506.441665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2506.510977] alg: No test for adler32 (adler32-zlib) [ 2507.422828] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 2507.594168] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2509.223247] Key type lgssc registered [ 2509.862095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2513.950596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2513.979076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.222812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.399099] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2521.753183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.972376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2523.997873] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2527.615847] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2530.335352] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2533.562158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2533.744544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.755246] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2539.879886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2540.139850] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2550.239929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2550.243663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2550.252953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2552.288899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2552.291778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2552.298056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.014691] Lustre: server umount lustre-OST0000 complete [ 2557.920155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2557.925215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.932331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2557.935916] Lustre: Skipped 1 previous similar message [ 2562.528642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2562.529198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.648394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2567.653723] Lustre: Skipped 2 previous similar messages [ 2572.255155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2572.346945] Lustre: server umount lustre-MDT0000 complete [ 2572.771068] LustreError: 103701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.781824] LustreError: 103701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2574.270600] LustreError: 103687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187953 with bad export cookie 16197093510240152293 [ 2574.274919] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2574.276575] LustreError: 103687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2574.450469] Lustre: server umount lustre-MDT0001 complete [ 2580.009774] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2581.630764] Key type lgssc unregistered [ 2581.818678] LNet: 107236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2581.823746] LNetError: 107236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2581.835258] LNet: Removed LNI 192.168.204.154@tcp [ 2582.283837] Key type .llcrypt unregistered [ 2582.287879] Key type ._llcrypt unregistered [ 2592.564580] Key type ._llcrypt registered [ 2592.565964] Key type .llcrypt registered [ 2592.631831] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2593.057437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2593.121745] alg: No test for adler32 (adler32-zlib) [ 2594.043131] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 2594.171616] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2595.799534] Key type lgssc registered [ 2596.365287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2600.121952] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2600.130764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2615.583693] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2620.704384] LustreError: 108451:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 2647.327556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.406189] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2654.510458] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2658.617862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.796822] LustreError: MGC192.168.204.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2658.809450] LustreError: 109475:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2658.815550] LustreError: 109475:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2658.820108] LustreError: 109475:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2658.867428] Lustre: server umount lustre-MDT0000 complete [ 2658.870710] LustreError: 109475:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2673.169471] Lustre: server umount lustre-OST0000 complete [ 2678.036825] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2682.651313] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2699.720261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.836180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2699.851632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2699.893855] Lustre: lustre-MDT0000: new disk, initializing [ 2699.933104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.941855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2701.847830] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2707.115472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.168373] Lustre: 112261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2707.203335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2707.205801] Lustre: Skipped 1 previous similar message [ 2707.248554] Lustre: lustre-MDT0001: new disk, initializing [ 2707.296076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2707.312109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2707.320087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2709.166790] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2714.222895] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2717.076148] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2720.608902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2720.765754] Lustre: lustre-OST0000: new disk, initializing [ 2720.769362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2720.777399] Lustre: 114229:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2720.818913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2723.844702] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2728.450512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2728.458893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2728.532481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2728.885442] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2730.941249] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2731.035244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2732.960072] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2733.065684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2738.656357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2738.657929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.662456] LustreError: Skipped 1 previous similar message [ 2738.666301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2738.672443] Lustre: Skipped 1 previous similar message [ 2743.779265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2743.785028] Lustre: Skipped 2 previous similar messages [ 2748.383133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2748.459424] Lustre: server umount lustre-OST0000 complete [ 2753.504596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2753.508829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.520356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2756.485075] Lustre: server umount lustre-MDT0000 complete [ 2758.655782] LustreError: 112252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188137 with bad export cookie 16385114474774475326 [ 2758.660553] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2758.661176] LustreError: 112252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2758.864035] Lustre: server umount lustre-MDT0001 complete [ 2763.139348] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 00:42:20 (1789188140) [ 2764.771749] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2772.414414] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2776.531441] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2791.545528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.669370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2791.685970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2791.726689] Lustre: lustre-MDT0000: new disk, initializing [ 2791.765589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.773252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2793.601225] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2799.011693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.076675] Lustre: 119121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2799.115784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2799.118835] Lustre: Skipped 1 previous similar message [ 2799.162378] Lustre: lustre-MDT0001: new disk, initializing [ 2799.228103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2799.234604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2800.943371] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2805.754555] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2808.223349] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2811.163775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.293158] Lustre: lustre-OST0000: new disk, initializing [ 2811.297221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2811.301927] Lustre: 121088:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2811.345142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.347856] Lustre: Skipped 1 previous similar message [ 2813.734948] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2818.048095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2818.053716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2818.108805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2818.367792] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2820.430439] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2820.528081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2822.450285] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2822.566423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2828.258432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.259520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2828.260087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2828.260091] Lustre: Skipped 2 previous similar messages [ 2828.264773] Lustre: Skipped 2 previous similar messages [ 2828.278714] LustreError: Skipped 1 previous similar message [ 2829.736832] Lustre: server umount lustre-OST0000 complete [ 2833.377039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2833.378298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2833.382672] Lustre: Skipped 1 previous similar message [ 2833.387287] Lustre: Skipped 2 previous similar messages [ 2837.624909] Lustre: server umount lustre-MDT0000 complete [ 2839.273081] LustreError: 119116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188218 with bad export cookie 16385114474774476194 [ 2839.275907] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2839.278364] LustreError: 119116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2839.414946] Lustre: server umount lustre-MDT0001 complete [ 2843.402232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.649406] LustreError: 123302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2843.661361] LustreError: 123302:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2843.693126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2845.498910] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2848.736129] LustreError: 123303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2849.308609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.300607] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2854.739425] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2857.213053] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2860.239445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2863.078896] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2866.532684] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2879.600837] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2884.064599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2884.067352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.072748] LustreError: Skipped 1 previous similar message [ 2884.085322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2887.513229] Lustre: server umount lustre-OST0000 complete [ 2895.264614] Lustre: server umount lustre-MDT0000 complete [ 2895.329721] LustreError: 124030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.337916] LustreError: 124030:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2896.842539] LustreError: 123284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188275 with bad export cookie 16385114474774477062 [ 2896.845917] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2896.849672] LustreError: 123284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2896.988798] Lustre: server umount lustre-MDT0001 complete [ 2901.724246] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2903.241978] Key type lgssc unregistered [ 2903.397617] LNet: 126949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2903.402291] LNetError: 126949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2903.416578] LNet: Removed LNI 192.168.204.154@tcp [ 2903.836607] Key type .llcrypt unregistered [ 2903.839103] Key type ._llcrypt unregistered [ 2911.172569] Key type ._llcrypt registered [ 2911.174863] Key type .llcrypt registered [ 2911.219227] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2917.965788] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2918.304263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2918.334590] alg: No test for adler32 (adler32-zlib) [ 2919.209569] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 2919.323973] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2920.919238] Key type lgssc registered [ 2921.390852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2936.874339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2936.882451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.977983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2937.992699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2938.032698] Lustre: lustre-MDT0000: new disk, initializing [ 2938.070613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2938.081153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2939.771939] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2944.988923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.048100] Lustre: 130071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2945.090212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2945.093550] Lustre: Skipped 1 previous similar message [ 2945.141401] Lustre: lustre-MDT0001: new disk, initializing [ 2945.186840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.204467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2945.211442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2947.001509] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2951.471933] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2953.640350] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2956.580043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2956.742678] Lustre: lustre-OST0000: new disk, initializing [ 2956.745622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2956.749539] Lustre: 132038:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2956.793158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.334242] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2964.150804] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2966.039833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2966.045865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2966.098701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2966.102652] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2966.198332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2968.021344] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2968.134555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2971.108544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2971.108756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.118489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2971.123034] Lustre: Skipped 1 previous similar message [ 2975.006906] Lustre: server umount lustre-OST0000 complete [ 2981.345160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.345267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2981.345930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2981.350119] Lustre: Skipped 2 previous similar messages [ 2981.353675] LustreError: Skipped 1 previous similar message [ 2982.567313] Lustre: server umount lustre-MDT0000 complete [ 2984.064447] LustreError: 130062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188362 with bad export cookie 7765784598996452032 [ 2984.069286] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2984.070047] LustreError: 130062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2984.192845] Lustre: server umount lustre-MDT0001 complete [ 2987.450359] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 00:46:04 (1789188364) [ 2990.113332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.345608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2992.109842] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2995.681362] LustreError: 134445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.689270] LustreError: 134445:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2995.893554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2996.067274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2997.893836] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3001.142820] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3003.334201] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3006.176884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.183235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3006.185732] Lustre: Skipped 2 previous similar messages [ 3010.268412] Lustre: server umount lustre-MDT0000 complete [ 3011.553318] LustreError: 134444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3011.560895] LustreError: 134444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3011.605789] LustreError: 136012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188390 with bad export cookie 7765784598996452900 [ 3011.608641] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3011.611278] LustreError: 136012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3011.749938] Lustre: server umount lustre-MDT0001 complete [ 3014.871536] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 00:46:32 (1789188392) [ 3017.786620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3033.247729] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3038.367485] LustreError: 136856:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 3064.031469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3066.616220] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3069.755107] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3070.667920] Lustre: server umount lustre-OST0000 complete [ 3074.043120] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 00:47:31 (1789188451) [ 3076.721276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.933524] LustreError: 138298:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.622508] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3082.174443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.208821] LustreError: 138299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.347278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3082.350628] Lustre: Skipped 1 previous similar message [ 3083.958689] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3086.735835] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3088.884356] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3091.453272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3094.046278] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3097.063420] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3107.298460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.309304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3107.315848] Lustre: Skipped 3 previous similar messages [ 3107.318974] Lustre: Skipped 1 previous similar message [ 3110.118849] Lustre: server umount lustre-MDT0000 complete [ 3111.657378] LustreError: 140140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188490 with bad export cookie 7765784598996453551 [ 3111.661121] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3111.664038] LustreError: 140140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3111.807204] Lustre: server umount lustre-MDT0001 complete [ 3122.395676] Lustre: server umount lustre-OST0000 complete [ 3125.558893] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 00:48:23 (1789188503) [ 3128.113435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3128.311317] LustreError: 141946:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.345485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3128.348148] Lustre: Skipped 1 previous similar message [ 3130.054523] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3133.386969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3135.172473] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3138.158232] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3140.519825] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3143.568994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3146.179890] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3149.468970] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3151.401391] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3153.961062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3154.593333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3156.364973] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3156.459042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3159.011407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.011617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3159.017368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3159.025994] Lustre: Skipped 2 previous similar messages [ 3163.418702] Lustre: server umount lustre-OST0000 complete [ 3169.248472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3169.249480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3169.260903] Lustre: Skipped 2 previous similar messages [ 3171.042656] Lustre: server umount lustre-MDT0000 complete [ 3172.615333] LustreError: 145277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188551 with bad export cookie 7765784598996454608 [ 3172.619433] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.625030] LustreError: 145277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3172.756603] Lustre: server umount lustre-MDT0001 complete [ 3176.091749] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 00:49:13 (1789188553) [ 3179.110798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3194.591659] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3199.711764] LustreError: 146121:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 3225.311309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.314964] Lustre: Skipped 2 previous similar messages [ 3227.654299] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3230.387408] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3233.076745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.253231] LustreError: 146146:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3233.263224] LustreError: 146146:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3233.319314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3234.862421] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3238.129940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.890703] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3242.769758] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3244.733140] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3246.296516] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3246.385530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3248.012316] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3248.095049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3249.633217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.634122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3249.637770] Lustre: Skipped 2 previous similar messages [ 3249.639745] Lustre: Skipped 2 previous similar messages [ 3254.998278] Lustre: server umount lustre-MDT0000 complete [ 3256.377128] LustreError: 146128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188635 with bad export cookie 7765784598996455616 [ 3256.380470] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3256.381979] LustreError: 146128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3256.499914] Lustre: server umount lustre-MDT0001 complete [ 3268.060965] Lustre: server umount lustre-OST0000 complete [ 3271.603515] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 00:50:49 (1789188649) [ 3274.258092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3289.695523] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3294.815372] LustreError: 150309:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 3322.578521] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3325.209164] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3327.677625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.852099] LustreError: 150334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3327.915648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3329.489078] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3333.057212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.857272] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3337.636476] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3339.668032] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3342.051714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3342.123952] Lustre: lustre-OST0001: new disk, initializing [ 3342.126157] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3342.129631] Lustre: 153180:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3342.482705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3342.487740] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3342.522786] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3344.306685] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3348.311460] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3349.855229] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3349.932799] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3351.372197] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3351.449788] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3353.057211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.061141] Lustre: Skipped 1 previous similar message [ 3353.061256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3353.064225] Lustre: Skipped 4 previous similar messages [ 3366.367093] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3366.397350] Lustre: server umount lustre-OST0000 complete [ 3368.419299] LustreError: 150324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.419741] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3368.423980] LustreError: 150324:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3368.426727] LustreError: Skipped 1 previous similar message [ 3373.770232] Lustre: server umount lustre-OST0001 complete [ 3381.146241] Lustre: server umount lustre-MDT0000 complete [ 3382.430532] LustreError: 152081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188761 with bad export cookie 7765784598996456568 [ 3382.433535] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3382.436431] LustreError: 152081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3382.562237] Lustre: server umount lustre-MDT0001 complete [ 3390.711807] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 00:52:48 (1789188768) [ 3391.340567] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3391.886545] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3392.470057] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 00:52:50 (1789188770) [ 3394.803044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.914260] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3394.928597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3395.008714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3395.010956] Lustre: Skipped 6 previous similar messages [ 3396.418858] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3399.403654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.442537] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3399.454288] Lustre: 156738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3401.015844] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3403.764329] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3405.793916] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3408.233814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3408.302461] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3410.325677] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3412.908163] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3414.499093] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3414.539919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3415.591448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3416.944852] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3417.020438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3419.617952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3419.619559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.621277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3419.624516] Lustre: Skipped 5 previous similar messages [ 3419.628826] Lustre: Skipped 11 previous similar messages [ 3423.763926] Lustre: server umount lustre-OST0000 complete [ 3437.764291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3439.911630] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3439.916233] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3439.916790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3440.026646] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3442.467689] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3444.029358] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3444.098632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3445.514454] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3445.588502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3447.443771] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9220435a6800.ost_server_uuid 50 [ 3447.958674] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9220435a6800.ost_server_uuid in FULL state after 0 sec [ 3448.013089] LustreError: 160008:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.54@tcp) returned error from blocking AST (req@00000000023ff2aa x1876099908955904 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d071b816/0x6bc59bf2a9621dc4 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.54@tcp remote: 0x3a53a61b6d69dc0d expref: 5 pid: 156746 timeout: 3548 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3448.025657] LustreError: lustre-MDT0000: A client on nid 192.168.204.54@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3448.028823] LustreError: 156734:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.54@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d071b816/0x6bc59bf2a9621dc4 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.54@tcp remote: 0x3a53a61b6d69dc0d expref: 6 pid: 156746 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3450.337021] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3450.339944] LustreError: Skipped 1 previous similar message [ 3455.116593] Lustre: server umount lustre-OST0000 complete [ 3462.482787] Lustre: server umount lustre-MDT0000 complete [ 3463.788808] LustreError: 156730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188842 with bad export cookie 7765784598996457751 [ 3463.791741] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3463.792764] LustreError: 156730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3469.938422] Lustre: server umount lustre-MDT0001 complete [ 3473.893712] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3475.031600] Key type lgssc unregistered [ 3475.164303] LNet: 162852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3475.166699] LNetError: 162852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3475.173278] LNet: Removed LNI 192.168.204.154@tcp [ 3475.457605] Key type .llcrypt unregistered [ 3475.458787] Key type ._llcrypt unregistered [ 3485.248749] Key type ._llcrypt registered [ 3485.251139] Key type .llcrypt registered [ 3485.307142] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3485.652515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3485.709026] alg: No test for adler32 (adler32-zlib) [ 3486.583214] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 3486.676564] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3488.255270] Key type lgssc registered [ 3488.688108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3491.515804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3491.523666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.673139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3494.102316] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3497.027305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.162067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3498.682967] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3501.355607] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3503.338844] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3505.658478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3505.790343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.837896] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3510.367525] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3518.118744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3523.234777] Lustre: Failing over lustre-MDT0000 [ 3523.369295] Lustre: server umount lustre-MDT0000 complete [ 3523.553228] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3523.553813] LustreError: 164231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.556928] Lustre: Skipped 1 previous similar message [ 3523.562422] LustreError: 164231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3528.159748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3528.162473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3528.166916] LustreError: 166374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.793033] LustreError: 164237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.798465] LustreError: 164237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3534.206488] LustreError: 166834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.54@tcp arrived at 1789188912 with bad export cookie 14851310765821103700 [ 3534.211373] LustreError: 166834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3536.578745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3536.620517] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3536.706275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.731217] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3538.078584] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3538.810054] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3538.812951] Lustre: lustre-MDT0000: Denying connection for new client 2d451f0f-a24e-4058-a22c-786f67e2000f (at 192.168.204.54@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3541.992431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3547.106736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3547.112268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3547.114471] Lustre: Skipped 1 previous similar message [ 3551.690131] Lustre: server umount lustre-OST0000 complete [ 3552.970347] LustreError: 167984:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3552.973375] Lustre: 167333:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3552.976064] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3552.981779] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3552.985900] LustreError: 167333:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000087375d9e x1876100503342720/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3552.994988] LustreError: 167333:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3552.999433] LustreError: 167333:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3553.003291] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 3553.102484] Lustre: server umount lustre-MDT0000 complete [ 3554.416965] LustreError: 164218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188933 with bad export cookie 14851310765821104029 [ 3554.421327] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3554.422827] LustreError: 164218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3554.426995] LustreError: 163547:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3560.622766] Lustre: server umount lustre-MDT0001 complete [ 3564.333994] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3565.446190] Key type lgssc unregistered [ 3565.569692] LNet: 168735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3565.573692] LNetError: 168735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3565.581632] LNet: Removed LNI 192.168.204.154@tcp [ 3565.905361] Key type .llcrypt unregistered [ 3565.907083] Key type ._llcrypt unregistered [ 3575.419143] Key type ._llcrypt registered [ 3575.420744] Key type .llcrypt registered [ 3575.468263] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3575.850599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3575.878080] alg: No test for adler32 (adler32-zlib) [ 3576.746696] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 3576.853378] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3578.447226] Key type lgssc registered [ 3578.825260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3581.526127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3581.533279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3582.693647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3583.995901] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3586.866585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.991413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3588.375913] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3590.736623] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3592.548226] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3594.876056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3594.989340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3597.081163] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3599.594259] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3607.273509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3612.641802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.647873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3612.648137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3612.654073] Lustre: Skipped 1 previous similar message [ 3613.523898] Lustre: server umount lustre-OST0000 complete [ 3616.976234] Lustre: server umount lustre-MDT0000 complete [ 3617.760419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3617.760837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3617.764822] LustreError: 171553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3617.768614] Lustre: Skipped 2 previous similar messages [ 3617.777844] LustreError: 171553:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3618.252225] LustreError: 170101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188997 with bad export cookie 8335022606705504759 [ 3618.256217] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3618.257120] LustreError: 170101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3618.365437] Lustre: server umount lustre-MDT0001 complete [ 3621.991265] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3623.085827] Key type lgssc unregistered [ 3623.216345] LNet: 173668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3623.218758] LNetError: 173668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3623.232240] LNet: Removed LNI 192.168.204.154@tcp [ 3623.549732] Key type .llcrypt unregistered [ 3623.551495] Key type ._llcrypt unregistered [ 3635.398870] Key type ._llcrypt registered [ 3635.399879] Key type .llcrypt registered [ 3635.433789] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3635.751265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3635.798363] alg: No test for adler32 (adler32-zlib) [ 3636.646622] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 3636.727290] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3638.303218] Key type lgssc registered [ 3638.659524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3641.275331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3641.282749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.423315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3643.768728] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3646.786313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.928422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3648.314666] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3650.703498] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3652.522334] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3654.825816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3654.937563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3656.848401] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3659.255518] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3668.264276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3672.545235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3672.548614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.555101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.571196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.571763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.572578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3677.012703] Lustre: server umount lustre-OST0000 complete [ 3678.688131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.688736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3678.693664] Lustre: Skipped 1 previous similar message [ 3678.695994] Lustre: Skipped 1 previous similar message [ 3683.808810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.811826] Lustre: Skipped 1 previous similar message [ 3684.438507] Lustre: server umount lustre-MDT0000 complete [ 3685.645141] LustreError: 175324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189064 with bad export cookie 6885332894141793517 [ 3685.648413] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3685.649627] LustreError: 175324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3685.761560] Lustre: server umount lustre-MDT0001 complete [ 3689.398309] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3690.388664] Key type lgssc unregistered [ 3690.530426] LNet: 178891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3690.533153] LNetError: 178891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3690.542362] LNet: Removed LNI 192.168.204.154@tcp [ 3690.832953] Key type .llcrypt unregistered [ 3690.834052] Key type ._llcrypt unregistered [ 3699.385878] Key type ._llcrypt registered [ 3699.386993] Key type .llcrypt registered [ 3699.433328] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3699.820807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3699.841679] alg: No test for adler32 (adler32-zlib) [ 3700.694499] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 3700.796827] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3702.383173] Key type lgssc registered [ 3702.758566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3706.264233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3706.271862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.423374] Lustre: *** cfs_fail_loc=135, val=0*** [ 3707.461561] LustreError: 180399:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3707.465255] LustreError: 180399:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -2 [ 3707.469795] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3707.469795] [ 3707.474099] LustreError: MGC192.168.204.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3707.480421] LustreError: 180361:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3707.484041] LustreError: 180361:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3707.487842] LustreError: 180361:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3707.490778] LustreError: 180361:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3707.522946] Lustre: server umount lustre-MDT0000 complete [ 3707.524103] LustreError: 180361:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3709.876658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.010713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.526502] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3714.676914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.859510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3716.454069] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3719.174276] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3721.117664] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3724.896844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3724.897548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3724.901617] Lustre: Skipped 1 previous similar message [ 3727.958919] Lustre: server umount lustre-MDT0000 complete [ 3729.297777] LustreError: 180751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189108 with bad export cookie 11808502212806583171 [ 3729.302894] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3729.303328] LustreError: 180751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3729.425311] Lustre: server umount lustre-MDT0001 complete [ 3733.296695] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3734.412636] Key type lgssc unregistered [ 3734.561460] LNet: 183089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3734.564974] LNetError: 183089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3734.574391] LNet: Removed LNI 192.168.204.154@tcp [ 3734.911774] Key type .llcrypt unregistered [ 3734.913589] Key type ._llcrypt unregistered [ 3742.615882] Key type ._llcrypt registered [ 3742.616991] Key type .llcrypt registered [ 3742.656703] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3742.766348] Key type .llcrypt unregistered [ 3742.767830] Key type ._llcrypt unregistered [ 3750.438579] Key type ._llcrypt registered [ 3750.440313] Key type .llcrypt registered [ 3750.481449] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3750.950296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3751.023257] alg: No test for adler32 (adler32-zlib) [ 3751.920929] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 3752.026575] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3753.615275] Key type lgssc registered [ 3754.030792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3756.783621] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3756.789612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3772.255446] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3777.375405] LustreError: 185189:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 3803.999182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3805.968040] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3808.404414] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3810.610625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.774763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3810.796345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3812.087159] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3814.931646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.051952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3816.373077] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3818.855845] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3820.742534] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3821.706836] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3829.730537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.731455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.734831] Lustre: Skipped 1 previous similar message [ 3832.595478] Lustre: server umount lustre-OST0000 complete [ 3834.848669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3834.849288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3834.853146] Lustre: Skipped 1 previous similar message [ 3834.854569] Lustre: Skipped 1 previous similar message [ 3839.968399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.970364] Lustre: Skipped 2 previous similar messages [ 3845.088291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3845.090407] Lustre: Skipped 1 previous similar message [ 3848.159162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3848.220924] Lustre: server umount lustre-MDT0000 complete [ 3849.427263] LustreError: 186255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189228 with bad export cookie 18407026667395990445 [ 3849.431196] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3849.432654] LustreError: 186255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3849.544811] Lustre: server umount lustre-MDT0001 complete [ 3852.956244] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3854.018677] Key type lgssc unregistered [ 3854.145330] LNet: 189103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3854.147774] LNetError: 189103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3854.158284] LNet: Removed LNI 192.168.204.154@tcp [ 3854.454863] Key type .llcrypt unregistered [ 3854.456231] Key type ._llcrypt unregistered [ 3862.802237] Key type ._llcrypt registered [ 3862.803405] Key type .llcrypt registered [ 3862.837983] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3863.181620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3863.252562] alg: No test for adler32 (adler32-zlib) [ 3864.106166] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 3864.198939] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3865.783267] Key type lgssc registered [ 3866.129409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3868.644371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3868.649596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3869.782301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3871.062856] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3873.678280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.783653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3875.024825] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3877.379290] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3879.169509] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3881.348950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3881.462561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3883.373697] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3885.760251] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3894.756736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3897.512829] Lustre: Failing over lustre-MDT0000 [ 3897.605978] Lustre: server umount lustre-MDT0000 complete [ 3899.359935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3899.363492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3899.369499] LustreError: 190487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.375681] LustreError: 190487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3899.872778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3899.873734] LustreError: 192987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.876662] Lustre: Skipped 1 previous similar message [ 3901.505694] LustreError: 191215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3901.511326] LustreError: 191215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3904.992675] LustreError: 190488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3904.997142] LustreError: 190488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3910.112587] LustreError: 190488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3910.117065] LustreError: 190488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3910.338807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.379425] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3910.465954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.494047] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3911.737606] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3911.745238] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3913.167485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3915.752893] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3915.762595] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3915.779208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3917.606755] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3920.865035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3920.866615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3920.868762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3920.872542] Lustre: Skipped 1 previous similar message [ 3920.876247] Lustre: Skipped 1 previous similar message [ 3925.771726] Lustre: server umount lustre-OST0000 complete [ 3931.105213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3931.105962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.112919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3931.114644] Lustre: Skipped 1 previous similar message [ 3936.223993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3936.225769] Lustre: Skipped 1 previous similar message [ 3940.320057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3940.322158] Lustre: Skipped 1 previous similar message [ 3941.343090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3941.404986] Lustre: server umount lustre-MDT0000 complete [ 3942.593384] LustreError: 194338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189321 with bad export cookie 2956538051878644796 [ 3942.596188] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3942.597952] LustreError: 194338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3942.701214] Lustre: server umount lustre-MDT0001 complete [ 3946.126701] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3947.121653] Key type lgssc unregistered [ 3947.242329] LNet: 195394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3947.244801] LNetError: 195394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3947.254283] LNet: Removed LNI 192.168.204.154@tcp [ 3947.563165] Key type .llcrypt unregistered [ 3947.564471] Key type ._llcrypt unregistered [ 3956.154070] Key type ._llcrypt registered [ 3956.155520] Key type .llcrypt registered [ 3956.190019] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3956.503756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3956.541523] alg: No test for adler32 (adler32-zlib) [ 3957.392298] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 3957.475818] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3959.055194] Key type lgssc registered [ 3959.417563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3961.942774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3961.947984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.073457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.454545] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3967.149677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3967.263145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3968.555531] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3970.990544] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3972.718309] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3974.939299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3975.088050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.965865] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3979.271205] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3987.373079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3990.145426] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3995.630179] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4008.365570] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4018.147152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.147185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4018.149472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4018.151020] Lustre: Skipped 1 previous similar message [ 4023.265087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4023.267096] Lustre: Skipped 2 previous similar messages [ 4023.562082] Lustre: server umount lustre-OST0000 complete [ 4028.385197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4028.385992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4028.390949] Lustre: Skipped 1 previous similar message [ 4028.394918] Lustre: Skipped 1 previous similar message [ 4033.504056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4033.505956] Lustre: Skipped 1 previous similar message [ 4038.624023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4038.626035] Lustre: Skipped 1 previous similar message [ 4039.135095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4039.197412] Lustre: server umount lustre-MDT0000 complete [ 4040.381486] LustreError: 196760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189419 with bad export cookie 10983013614091897597 [ 4040.387109] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4040.503318] Lustre: server umount lustre-MDT0001 complete [ 4044.285000] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4045.259742] Key type lgssc unregistered [ 4045.378341] LNet: 200663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4045.381256] LNetError: 200663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4045.388332] LNet: Removed LNI 192.168.204.154@tcp [ 4045.700838] Key type .llcrypt unregistered [ 4045.702104] Key type ._llcrypt unregistered [ 4052.873659] Key type ._llcrypt registered [ 4052.875162] Key type .llcrypt registered [ 4052.912217] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4053.473552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4053.484297] alg: No test for adler32 (adler32-zlib) [ 4054.338765] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 4054.431328] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4056.007185] Key type lgssc registered [ 4056.349433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4058.948633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4058.955637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.105851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4061.368877] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4064.126918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.247839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4065.612974] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4067.887554] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4069.634755] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4071.819232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4071.921517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4073.751735] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4076.183579] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4085.221796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4088.172495] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4095.804901] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4102.465546] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4108.814090] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4114.911960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4114.914924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.920930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4115.936565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4115.936791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4115.937097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.945777] Lustre: Skipped 1 previous similar message [ 4121.056760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4121.060339] Lustre: Skipped 1 previous similar message [ 4126.176691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4126.179140] Lustre: Skipped 1 previous similar message [ 4128.223179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4128.278744] Lustre: server umount lustre-OST0000 complete [ 4130.783810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4130.786347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.792180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4135.643473] Lustre: server umount lustre-MDT0000 complete [ 4136.415964] LustreError: 203333:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4136.421300] LustreError: 203333:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4136.890098] LustreError: 201881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189515 with bad export cookie 5113240592765608409 [ 4136.893820] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4136.894190] LustreError: 201881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4137.009091] Lustre: server umount lustre-MDT0001 complete [ 4140.515472] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4141.510803] Key type lgssc unregistered [ 4141.642362] LNet: 207132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4141.644661] LNetError: 207132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4141.655405] LNet: Removed LNI 192.168.204.154@tcp [ 4141.951712] Key type .llcrypt unregistered [ 4141.952789] Key type ._llcrypt unregistered [ 4150.443094] Key type ._llcrypt registered [ 4150.445232] Key type .llcrypt registered [ 4150.480515] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4150.789933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4150.820823] alg: No test for adler32 (adler32-zlib) [ 4151.665661] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 4151.757144] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4153.335161] Key type lgssc registered [ 4153.675879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4155.979787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4155.987284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4157.119611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.392620] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4161.070563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.191956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4162.487943] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4164.861158] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4166.621107] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4168.829292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4168.965397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4170.841755] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4173.202636] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4181.285835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4191.713887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.715428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4191.718775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4191.720532] LustreError: Skipped 1 previous similar message [ 4191.722577] Lustre: Skipped 1 previous similar message [ 4192.860544] Lustre: server umount lustre-OST0000 complete [ 4196.832365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.836394] Lustre: Skipped 1 previous similar message [ 4196.838246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4196.840294] Lustre: Skipped 1 previous similar message [ 4200.285112] Lustre: server umount lustre-MDT0000 complete [ 4201.476423] LustreError: 208498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189580 with bad export cookie 2635966498773157227 [ 4201.478811] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4201.480352] LustreError: 208498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4201.588408] Lustre: server umount lustre-MDT0001 complete [ 4205.096658] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4206.181855] Key type lgssc unregistered [ 4206.314319] LNet: 212223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4206.316949] LNetError: 212223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4206.328308] LNet: Removed LNI 192.168.204.154@tcp [ 4206.659653] Key type .llcrypt unregistered [ 4206.660959] Key type ._llcrypt unregistered [ 4214.231323] Key type ._llcrypt registered [ 4214.232627] Key type .llcrypt registered [ 4214.272800] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4214.760802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4214.776039] alg: No test for adler32 (adler32-zlib) [ 4215.628095] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 4215.735818] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4217.327234] Key type lgssc registered [ 4217.719916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4220.245844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4220.252234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.402860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.697259] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4225.329595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.451612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4226.737467] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4228.989202] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4230.680544] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4232.812633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4232.926934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.773287] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4237.133963] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4246.252118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4251.616648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4251.617284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.618701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4251.619456] LustreError: Skipped 1 previous similar message [ 4251.623360] Lustre: Skipped 1 previous similar message [ 4252.948063] Lustre: server umount lustre-OST0000 complete [ 4256.224142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4256.227583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4256.233565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4256.235423] Lustre: Skipped 1 previous similar message [ 4256.290594] Lustre: server umount lustre-MDT0000 complete [ 4256.737349] LustreError: 213499:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4256.743018] LustreError: 213499:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4257.514234] LustreError: 213457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189636 with bad export cookie 10904175128952400419 [ 4257.516720] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4257.518675] LustreError: 213457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4257.645726] Lustre: server umount lustre-MDT0001 complete [ 4261.186963] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4262.202034] Key type lgssc unregistered [ 4262.335384] LNet: 217170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4262.338448] LNetError: 217170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4262.348295] LNet: Removed LNI 192.168.204.154@tcp [ 4262.655321] Key type .llcrypt unregistered [ 4262.656766] Key type ._llcrypt unregistered [ 4269.834706] Key type ._llcrypt registered [ 4269.835677] Key type .llcrypt registered [ 4269.867802] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4270.223052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4270.238912] alg: No test for adler32 (adler32-zlib) [ 4271.100446] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 4271.194859] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4272.775185] Key type lgssc registered [ 4273.154410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4275.744269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4275.750484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4276.921170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.200681] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4280.914718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4281.027548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4282.370741] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4284.737224] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4286.402196] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4288.529465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4288.636828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4290.396934] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4292.809778] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4303.018025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4305.798505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4305.824077] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4305.905700] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4307.865051] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4310.066059] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4310.576539] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4321.163860] Lustre: Permanently deactivating lustre-OST0001 [ 4321.172512] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4321.523601] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4332.998265] Lustre: Permanently reactivating lustre-OST0001 [ 4333.002946] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4333.004987] Lustre: Skipped 2 previous similar messages [ 4340.727424] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4340.731647] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4340.733769] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4340.738541] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4340.752460] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4345.314343] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.316150] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4345.319268] Lustre: Skipped 2 previous similar messages [ 4345.322468] Lustre: Skipped 1 previous similar message [ 4347.198158] Lustre: server umount lustre-OST0001 complete [ 4349.407865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4349.411034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.416727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4350.432703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4350.434070] LustreError: 220225:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.439127] LustreError: 220225:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4354.508553] Lustre: server umount lustre-OST0000 complete [ 4358.111857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4358.113913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.116945] Lustre: Skipped 1 previous similar message [ 4358.119184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.120644] Lustre: Skipped 1 previous similar message [ 4361.881961] Lustre: server umount lustre-MDT0000 complete [ 4363.028123] LustreError: 218390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189741 with bad export cookie 11321701907930180109 [ 4363.030581] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4363.030908] LustreError: 218390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4363.152459] Lustre: server umount lustre-MDT0001 complete [ 4366.690138] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4367.706898] Key type lgssc unregistered [ 4367.831441] LNet: 223483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4367.834417] LNetError: 223483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4367.841370] LNet: Removed LNI 192.168.204.154@tcp [ 4368.174881] Key type .llcrypt unregistered [ 4368.176249] Key type ._llcrypt unregistered [ 4382.001559] Key type ._llcrypt registered [ 4382.002759] Key type .llcrypt registered [ 4382.036789] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4382.398292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4382.417253] alg: No test for adler32 (adler32-zlib) [ 4383.271096] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 4383.350905] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4384.927256] Key type lgssc registered [ 4385.330294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4387.987542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4387.995610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4389.072625] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4389.078729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4389.128386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4390.495404] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4393.399037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4393.431437] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4393.442751] Lustre: 225658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4393.532900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4394.846216] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4397.091184] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4398.822130] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4400.964468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4401.028641] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4401.086990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.314059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 4402.894396] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4405.166512] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4406.712345] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4414.228169] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4420.718938] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4429.258836] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4444.270651] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4444.272954] Lustre: Skipped 1 previous similar message [ 4462.805113] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4462.807557] Lustre: Skipped 2 previous similar messages [ 4501.476572] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4501.480199] Lustre: Skipped 4 previous similar messages [ 4565.473512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4565.475226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4565.476408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4565.482341] Lustre: Skipped 1 previous similar message [ 4568.721927] Lustre: server umount lustre-OST0000 complete [ 4570.592434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.593028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4570.596571] Lustre: Skipped 1 previous similar message [ 4570.598756] Lustre: Skipped 1 previous similar message [ 4575.712150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4575.714763] Lustre: Skipped 2 previous similar messages [ 4576.156642] Lustre: server umount lustre-MDT0000 complete [ 4577.442131] LustreError: 227517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189956 with bad export cookie 12708740867936357872 [ 4577.444850] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4577.445510] LustreError: 227517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4577.573378] Lustre: server umount lustre-MDT0001 complete [ 4581.362759] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4582.514509] Key type lgssc unregistered [ 4582.642584] LNet: 230457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4582.645620] LNetError: 230457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4582.655432] LNet: Removed LNI 192.168.204.154@tcp [ 4582.967044] Key type .llcrypt unregistered [ 4582.968058] Key type ._llcrypt unregistered [ 4591.296798] Key type ._llcrypt registered [ 4591.298201] Key type .llcrypt registered [ 4591.335384] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4591.634169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4591.680366] alg: No test for adler32 (adler32-zlib) [ 4592.519061] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 4592.597353] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4594.175171] Key type lgssc registered [ 4594.556432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4596.969815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4596.975076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4598.102346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4599.295346] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4601.903804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4602.026952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4603.279799] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4605.485212] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4607.200161] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4609.305030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4609.421587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4611.318298] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4613.591240] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4623.781792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 4628.962090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4628.962348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4628.962948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4628.965491] Lustre: Skipped 1 previous similar message [ 4629.984687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4629.987547] Lustre: Skipped 2 previous similar messages [ 4633.614458] Lustre: server umount lustre-OST0000 complete [ 4635.105036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.105603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4635.109449] Lustre: Skipped 1 previous similar message [ 4635.112778] Lustre: Skipped 1 previous similar message [ 4641.051947] Lustre: server umount lustre-MDT0000 complete [ 4641.249587] LustreError: 233339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4641.255356] LustreError: 233339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4642.414183] LustreError: 231822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190021 with bad export cookie 11031085134542547810 [ 4642.417239] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4642.418214] LustreError: 231822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4642.554453] Lustre: server umount lustre-MDT0001 complete [ 4646.200527] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4647.206382] Key type lgssc unregistered [ 4647.342465] LNet: 235552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4647.345253] LNetError: 235552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4647.352431] LNet: Removed LNI 192.168.204.154@tcp [ 4647.654832] Key type .llcrypt unregistered [ 4647.656551] Key type ._llcrypt unregistered [ 4656.230391] Key type ._llcrypt registered [ 4656.231611] Key type .llcrypt registered [ 4656.274078] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4656.802680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4656.831089] alg: No test for adler32 (adler32-zlib) [ 4657.697372] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 4657.799625] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4659.391206] Key type lgssc registered [ 4659.775487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4925.546752] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4926.584418] Key type lgssc unregistered [ 4926.731434] LNet: 237550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4926.733749] LNetError: 237550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4926.740368] LNet: Removed LNI 192.168.204.154@tcp [ 4927.092133] Key type .llcrypt unregistered [ 4927.093381] Key type ._llcrypt unregistered [ 4935.652198] Key type ._llcrypt registered [ 4935.653504] Key type .llcrypt registered [ 4935.689940] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4944.110862] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4944.829948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4944.852782] alg: No test for adler32 (adler32-zlib) [ 4945.821713] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 4945.986149] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4947.607269] Key type lgssc registered [ 4948.374544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4966.854814] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4671 sec ========== 01:19:03 (1789190343) [ 4968.171491] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:19:04 (1789190344) === [ 4970.893172] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:19:07 (1789190347) === [ 4989.357195] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4990.497369] Key type lgssc unregistered [ 4990.642345] LNet: 242751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4990.646121] LNetError: 242751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4990.657512] LNet: Removed LNI 192.168.204.154@tcp [ 4990.967323] Key type .llcrypt unregistered [ 4990.968969] Key type ._llcrypt unregistered