[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 454084949 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2608MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002247] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013061] *** VALIDATE selinux *** [ 0.021619] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026180] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029098] *** VALIDATE tmpfs *** [ 0.030432] *** VALIDATE proc *** [ 0.031211] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035102] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041054] debug: unmapping init [mem 0xffffffff86a59000-0xffffffff86a60fff] [ 0.043175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044620] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045017] ... version: 2 [ 0.046010] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051008] ... event mask: 000000070000000f [ 0.052235] rcu: Hierarchical SRCU implementation. [ 0.054039] smp: Bringing up secondary CPUs ... [ 0.055409] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.059287] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134662] node 0 deferred pages initialised in 70ms [ 0.137238] devtmpfs: initialized [ 0.138162] x86/mm: Memory block size: 128MB [ 0.140663] gcov: version magic: 0x41383552 [ 0.141610] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148330] pinctrl core: initialized pinctrl subsystem [ 0.150109] [ 0.150708] ************************************************************* [ 0.153010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155007] ** ** [ 0.157007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158007] ** ** [ 0.160012] ** This means that this kernel is built to expose internal ** [ 0.161009] ** IOMMU data structures, which may compromise security on ** [ 0.163011] ** your system. ** [ 0.165011] ** ** [ 0.167010] ** If you see this message and you are not debugging the ** [ 0.169010] ** kernel, report this immediately to your vendor! ** [ 0.172009] ** ** [ 0.173008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176011] ************************************************************* [ 0.178620] NET: Registered protocol family 16 [ 0.180420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.186054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.190013] cpuidle: using governor menu [ 0.191833] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194424] PCI: Using configuration type 1 for base access [ 0.196137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209028] cryptd: max_cpu_qlen set to 1000 [ 0.212271] ACPI: Added _OSI(Module Device) [ 0.213011] ACPI: Added _OSI(Processor Device) [ 0.214011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224677] ACPI: Interpreter enabled [ 0.226063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228011] ACPI: Using IOAPIC for interrupt routing [ 0.229092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.233397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245051] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258912] acpiphp: Slot [2] registered [ 0.260092] acpiphp: Slot [5] registered [ 0.261081] acpiphp: Slot [6] registered [ 0.262098] acpiphp: Slot [7] registered [ 0.264078] acpiphp: Slot [8] registered [ 0.265058] acpiphp: Slot [9] registered [ 0.267095] acpiphp: Slot [10] registered [ 0.268091] acpiphp: Slot [3] registered [ 0.269096] acpiphp: Slot [4] registered [ 0.271062] acpiphp: Slot [11] registered [ 0.272091] acpiphp: Slot [12] registered [ 0.273075] acpiphp: Slot [13] registered [ 0.275101] acpiphp: Slot [14] registered [ 0.276068] acpiphp: Slot [15] registered [ 0.277069] acpiphp: Slot [16] registered [ 0.278065] acpiphp: Slot [17] registered [ 0.280082] acpiphp: Slot [18] registered [ 0.281087] acpiphp: Slot [19] registered [ 0.282070] acpiphp: Slot [20] registered [ 0.284093] acpiphp: Slot [21] registered [ 0.285077] acpiphp: Slot [22] registered [ 0.286070] acpiphp: Slot [23] registered [ 0.287076] acpiphp: Slot [24] registered [ 0.288101] acpiphp: Slot [25] registered [ 0.289092] acpiphp: Slot [26] registered [ 0.291097] acpiphp: Slot [27] registered [ 0.292148] acpiphp: Slot [28] registered [ 0.293162] acpiphp: Slot [29] registered [ 0.294075] acpiphp: Slot [30] registered [ 0.295144] acpiphp: Slot [31] registered [ 0.297055] PCI host bridge to bus 0000:00 [ 0.298015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.300015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.302020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.304018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.309015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310148] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.313861] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.316251] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.325013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.329614] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.332012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.333012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.336015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.338555] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.341748] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.344041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.347790] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.352010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.363013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.367013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.373106] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.384015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.394029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.426014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.439625] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.445013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.454015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.472012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.482255] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.490014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.495012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.513014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.523865] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.533013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.539014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.556016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.566000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.574012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.582011] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.595013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.604575] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.611013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.617013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.635015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.645654] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.648414] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.651383] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.654380] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.657235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.661220] iommu: Default domain type: Passthrough [ 0.662424] SCSI subsystem initialized [ 0.664121] ACPI: bus type USB registered [ 0.665091] usbcore: registered new interface driver usbfs [ 0.667057] usbcore: registered new interface driver hub [ 0.669058] usbcore: registered new device driver usb [ 0.670139] pps_core: LinuxPPS API ver. 1 registered [ 0.672008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.675052] PTP clock support registered [ 0.677035] EDAC MC: Ver: 3.0.0 [ 0.679107] PCI: Using ACPI for IRQ routing [ 0.680420] NetLabel: Initializing [ 0.682013] NetLabel: domain hash size = 128 [ 0.683007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.685088] NetLabel: unlabeled traffic allowed by default [ 0.687138] vgaarb: loaded [ 0.688248] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.691013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.695518] clocksource: Switched to clocksource kvm-clock [ 0.788364] VFS: Disk quotas dquot_6.6.0 [ 0.790406] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.792633] *** VALIDATE ramfs *** [ 0.793649] *** VALIDATE hugetlbfs *** [ 0.795021] pnp: PnP ACPI init [ 0.797106] pnp: PnP ACPI: found 6 devices [ 0.812795] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.816151] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.818813] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.821281] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.824095] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.826571] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.829477] NET: Registered protocol family 2 [ 0.832033] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.836648] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.840510] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.845832] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.849099] TCP: Hash tables configured (established 65536 bind 65536) [ 0.851633] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.853955] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.856559] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.859443] NET: Registered protocol family 1 [ 0.861536] RPC: Registered named UNIX socket transport module. [ 0.863894] RPC: Registered udp transport module. [ 0.865757] RPC: Registered tcp transport module. [ 0.867772] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.870415] NET: Registered protocol family 44 [ 0.872934] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.874965] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.877282] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.880491] PCI: CLS 0 bytes, default 64 [ 0.883636] Unpacking initramfs... [ 2.297501] debug: unmapping init [mem 0xffff895d3cc54000-0xffff895d3ffbffff] [ 2.301352] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.303954] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.305938] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.776238] Initialise system trusted keyrings [ 2.777870] Key type blacklist registered [ 2.781512] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.793328] zbud: loaded [ 2.797483] *** VALIDATE nfs *** [ 2.798974] *** VALIDATE nfs4 *** [ 2.800893] pstore: using deflate compression [ 2.804323] Platform Keyring initialized [ 2.911596] NET: Registered protocol family 38 [ 2.913550] Key type asymmetric registered [ 2.914948] Asymmetric key parser 'x509' registered [ 2.917172] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.919665] io scheduler mq-deadline registered [ 2.921097] io scheduler kyber registered [ 2.922824] io scheduler bfq registered [ 2.924472] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.926918] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.929528] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.933511] ACPI: Power Button [PWRF] [ 2.939277] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.946122] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.959658] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.966649] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.985586] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.017494] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.048235] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.053791] Non-volatile memory driver v1.3 [ 3.056099] Linux agpgart interface v0.103 [ 3.088595] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.091227] vda: detected capacity change from 0 to 75071488 [ 3.105260] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.109460] vdb: detected capacity change from 0 to 1073741824 [ 3.123767] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.127028] vdc: detected capacity change from 0 to 2621440000 [ 3.141218] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.144135] vdd: detected capacity change from 0 to 2621440000 [ 3.156791] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.158926] vde: detected capacity change from 0 to 4294967296 [ 3.171580] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.174926] vdf: detected capacity change from 0 to 4294967296 [ 3.180601] libphy: Fixed MDIO Bus: probed [ 3.185202] usbcore: registered new interface driver usbserial_generic [ 3.187560] usbserial: USB Serial support registered for generic [ 3.189701] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.193262] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.194533] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.196972] mousedev: PS/2 mouse device common for all mice [ 3.199762] rtc_cmos 00:05: RTC can wake from S4 [ 3.201778] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.202710] rtc_cmos 00:05: registered as rtc0 [ 3.208406] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.210084] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.212113] intel_pstate: CPU model not supported [ 3.216930] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.220729] hid: raw HID events driver (C) Jiri Kosina [ 3.222488] usbcore: registered new interface driver usbhid [ 3.224270] usbhid: USB HID core driver [ 3.225545] drop_monitor: Initializing network drop monitor service [ 3.228016] Initializing XFRM netlink socket [ 3.229720] NET: Registered protocol family 10 [ 3.232549] Segment Routing with IPv6 [ 3.234275] NET: Registered protocol family 17 [ 3.235854] mpls_gso: MPLS GSO support [ 3.240770] RAS: Correctable Errors collector initialized. [ 3.243233] AVX version of gcm_enc/dec engaged. [ 3.245390] AES CTR mode by8 optimization enabled [ 3.323318] sched_clock: Marking stable (3323299042, 0)->(4178325850, -855026808) [ 3.326646] registered taskstats version 1 [ 3.329601] Loading compiled-in X.509 certificates [ 3.331536] zswap: loaded using pool lzo/zbud [ 3.351805] Key type big_key registered [ 3.362477] Key type encrypted registered [ 3.364127] ima: No TPM chip found, activating TPM-bypass! [ 3.365945] ima: Allocated hash algorithm: sha1 [ 3.367657] ima: No architecture policies found [ 3.369296] evm: Initialising EVM extended attributes: [ 3.371723] evm: security.selinux [ 3.372927] evm: security.ima [ 3.374118] evm: security.capability [ 3.375793] evm: HMAC attrs: 0x1 [ 3.377933] rtc_cmos 00:05: setting system clock to 2026-08-27 11:29:39 UTC (1787830179) [ 3.383396] debug: unmapping init [mem 0xffffffff87a03000-0xffffffff87bfffff] [ 3.385923] debug: unmapping init [mem 0xffffffff86782000-0xffffffff86a58fff] [ 3.394144] Write protecting the kernel read-only data: 28672k [ 3.397481] debug: unmapping init [mem 0xffffffff84e03000-0xffffffff84ffffff] [ 3.401317] debug: unmapping init [mem 0xffffffff85714000-0xffffffff857fffff] [ 3.437916] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.445354] systemd[1]: Detected virtualization kvm. [ 3.448162] systemd[1]: Detected architecture x86-64. [ 3.450123] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.478443] systemd[1]: No hostname configured. [ 3.479899] systemd[1]: Set hostname to . [ 3.482365] random: systemd: uninitialized urandom read (16 bytes read) [ 3.485264] systemd[1]: Initializing machine ID from random generator. [ 3.531307] random: ln: uninitialized urandom read (6 bytes read) [ 3.611516] random: systemd: uninitialized urandom read (16 bytes read) [ 3.614346] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.619065] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.624167] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.215113] device-mapper: uevent: version 1.0.3 [ 4.217522] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.637918] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.031653] virtio_net virtio0 ens2: renamed from eth0 [ 5.116030] scsi host0: ata_piix [ 5.149828] scsi host1: ata_piix [ 5.152693] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.155728] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.007569] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.880450] random: crng init done [ 9.881953] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.518537] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.796927] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.121674] SELinux: Disabled at runtime. [ 12.195118] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.210571] systemd[1]: Detected virtualization kvm. [ 12.213151] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.854090] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.861507] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.872065] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.876341] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.884314] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.896382] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.908638] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ 13.150771] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.549269] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.925141] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.994794] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.160654] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.174149] EDAC sbridge: Ver: 1.1.2 [ 16.448613] Key type dns_resolver registered [ 16.788245] NFS: Registering the id_resolver key type [ 16.790987] Key type id_resolver registered [ 16.794158] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg359-server login: [ 44.959341] libcfs: loading out-of-tree module taints kernel. [ 44.984711] Key type ._llcrypt registered [ 44.986852] Key type .llcrypt registered [ 45.039774] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 54.887613] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 55.553167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 55.565730] alg: No test for adler32 (adler32-zlib) [ 56.623946] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 57.026853] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 58.680404] Key type lgssc registered [ 59.430947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 67.239897] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 77.169065] hrtimer: interrupt took 4035862 ns [ 85.778225] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 91.832386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 91.844076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 92.973476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 92.990638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 93.035587] Lustre: lustre-MDT0000: new disk, initializing [ 93.090142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.103083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 94.870818] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.452786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 101.508926] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 101.529385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 101.534206] Lustre: Skipped 1 previous similar message [ 101.582084] Lustre: lustre-MDT0001: new disk, initializing [ 101.619337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.638249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 101.646737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 103.538960] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.416905] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 110.582230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 110.701222] Lustre: lustre-OST0000: new disk, initializing [ 110.703619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 110.707573] Lustre: 8421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 110.733657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 113.363755] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.786278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 113.791452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 113.834401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 120.201107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 120.258330] Lustre: lustre-OST0001: new disk, initializing [ 120.260851] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 120.264272] Lustre: 9494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 120.291948] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 123.045835] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 126.452366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 126.457148] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 126.487210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 130.142337] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 134.504500] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 137.060927] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing check_logdir /tmp/testlogs/ [ 139.063380] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing yml_node [ 141.322260] Lustre: DEBUG MARKER: Client: 2.17.57.61 [ 142.678252] Lustre: DEBUG MARKER: MDS: 2.17.57.61 [ 143.908476] Lustre: DEBUG MARKER: OSS: 2.17.57.61 [ 144.757885] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 07:32:01 EDT 2026 [ 153.219940] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 153.896409] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 167.393997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 167.395215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 167.403357] Lustre: Skipped 2 previous similar messages [ 167.407383] Lustre: Skipped 1 previous similar message [ 172.514374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 172.518493] Lustre: Skipped 5 previous similar messages [ 172.726772] Lustre: server umount lustre-MDT0000 complete [ 174.462443] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830350 with bad export cookie 2528005833744193937 [ 174.465483] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 174.471596] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 177.632910] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 177.636226] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 177.640187] Lustre: Skipped 2 previous similar messages [ 177.650645] Lustre: Skipped 1 previous similar message [ 179.681387] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 179.684790] Lustre: Skipped 1 previous similar message [ 180.663807] Lustre: server umount lustre-MDT0001 complete [ 182.312188] Lustre: server umount lustre-OST0000 complete [ 183.941958] Lustre: server umount lustre-OST0001 complete [ 186.469832] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 190.669208] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 207.394224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.500349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 207.519444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 207.573110] Lustre: lustre-MDT0000: new disk, initializing [ 207.617031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.624968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 209.519128] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 215.075130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.126412] Lustre: 15049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.257042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.990266] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 221.898609] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 224.444604] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 227.545287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 227.686071] Lustre: lustre-OST0000: new disk, initializing [ 227.688194] Lustre: Skipped 1 previous similar message [ 227.692315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 227.697554] Lustre: Skipped 2 previous similar messages [ 227.702593] Lustre: 17017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.745179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.748962] Lustre: Skipped 1 previous similar message [ 229.201902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.207575] Lustre: Skipped 1 previous similar message [ 229.211286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 229.274465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 230.581150] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 235.744266] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 237.714357] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 237.813252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 239.809554] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 239.907333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 244.707846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 244.708742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 244.713932] LustreError: Skipped 1 previous similar message [ 244.715134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 244.720491] Lustre: Skipped 1 previous similar message [ 244.730885] Lustre: Skipped 1 previous similar message [ 246.962546] Lustre: server umount lustre-OST0000 complete [ 249.827092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.946573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 254.950336] Lustre: Skipped 3 previous similar messages [ 263.136419] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 263.223157] Lustre: server umount lustre-MDT0000 complete [ 264.161611] LustreError: 15060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 264.170015] LustreError: 15060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 264.727247] LustreError: 15042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830440 with bad export cookie 2528005833744195456 [ 264.730053] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 264.732437] LustreError: 15042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 264.874301] Lustre: server umount lustre-MDT0001 complete [ 266.289994] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 07:34:02 (1787830442) [ 269.128040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 269.337351] LustreError: 19281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 269.367477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 270.970987] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 274.404821] LustreError: 19282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 274.709492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 276.523120] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 279.783844] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 282.229963] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 285.650940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 288.625215] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 291.974561] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 304.609736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 304.613253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.614760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 304.621069] Lustre: Skipped 2 previous similar messages [ 304.624434] Lustre: Skipped 2 previous similar messages [ 306.075401] Lustre: server umount lustre-OST0000 complete [ 314.198666] Lustre: server umount lustre-MDT0000 complete [ 314.849399] LustreError: 19282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.856501] LustreError: 19282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 316.238172] LustreError: 21881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830492 with bad export cookie 2528005833744196338 [ 316.243291] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.247057] LustreError: 21881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 316.423181] Lustre: server umount lustre-MDT0001 complete [ 322.961936] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 324.736755] Key type lgssc unregistered [ 324.974137] LNet: 22835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.981306] LNetError: 22835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 324.991451] LNet: Removed LNI 192.168.203.159@tcp [ 325.720152] Key type .llcrypt unregistered [ 325.722434] Key type ._llcrypt unregistered [ 338.714866] Key type ._llcrypt registered [ 338.720302] Key type .llcrypt registered [ 338.785549] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 339.326398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 339.348693] alg: No test for adler32 (adler32-zlib) [ 340.241947] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 340.392543] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 342.000462] Key type lgssc registered [ 342.607547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 346.832982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 346.855927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 348.084306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.908674] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 353.621992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 353.826909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 355.737432] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 359.043898] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 361.593518] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 365.570102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 365.787259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.798289] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 372.982884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 373.085165] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 383.457032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 383.460556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.461046] LustreError: Skipped 1 previous similar message [ 383.467823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 383.472573] Lustre: Skipped 1 previous similar message [ 384.236111] Lustre: server umount lustre-OST0000 complete [ 388.087324] Lustre: server umount lustre-MDT0000 complete [ 388.578577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.579254] LustreError: 24227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.584240] Lustre: Skipped 2 previous similar messages [ 388.594420] LustreError: 24227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 389.846559] LustreError: 24209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830565 with bad export cookie 9368228953438668614 [ 389.852561] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 389.854730] LustreError: 24209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 390.031704] Lustre: server umount lustre-MDT0001 complete [ 394.897725] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 396.300913] Key type lgssc unregistered [ 396.454420] LNet: 28202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.461245] LNetError: 28202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 396.472742] LNet: Removed LNI 192.168.203.159@tcp [ 396.862306] Key type .llcrypt unregistered [ 396.864296] Key type ._llcrypt unregistered [ 407.950646] Key type ._llcrypt registered [ 407.952331] Key type .llcrypt registered [ 408.007755] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 408.413899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 408.457721] alg: No test for adler32 (adler32-zlib) [ 409.398023] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 409.510115] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 411.112170] Key type lgssc registered [ 411.586523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 415.033583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 415.045651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.260777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.118277] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 422.177674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.380853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 424.220727] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 427.542968] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 430.283396] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 436.681115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 436.846050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.463684] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 440.942735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 442.770253] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 451.553760] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 451.557539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.571243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 452.583992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 452.586568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 452.588597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.636513] Lustre: server umount lustre-OST0000 complete [ 457.703711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.709144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 457.718563] Lustre: Skipped 1 previous similar message [ 457.723637] Lustre: Skipped 2 previous similar messages [ 461.959835] Lustre: server umount lustre-MDT0000 complete [ 462.821351] LustreError: 29592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.828579] LustreError: 29592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 463.826718] LustreError: 29573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830639 with bad export cookie 1635344327007038926 [ 463.829198] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 463.832743] LustreError: 29573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 464.005155] Lustre: server umount lustre-MDT0001 complete [ 469.760952] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 471.280956] Key type lgssc unregistered [ 471.451751] LNet: 33470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.458933] LNetError: 33470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 471.472729] LNet: Removed LNI 192.168.203.159@tcp [ 471.877203] Key type .llcrypt unregistered [ 471.878742] Key type ._llcrypt unregistered [ 484.708166] Key type ._llcrypt registered [ 484.711123] Key type .llcrypt registered [ 484.772564] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 485.309509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 485.342092] alg: No test for adler32 (adler32-zlib) [ 486.309828] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 486.477638] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 488.112308] Key type lgssc registered [ 488.890625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 493.707309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 493.718256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.989318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.600265] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 502.740823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.965953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 505.752312] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 509.494801] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 512.326087] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 515.657047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 515.857570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 518.905182] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 519.983154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 522.444071] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 533.475436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 533.477307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.482899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 533.492095] Lustre: Skipped 1 previous similar message [ 537.155964] Lustre: server umount lustre-OST0000 complete [ 543.715424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 543.717742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 543.727241] Lustre: Skipped 1 previous similar message [ 543.730637] Lustre: Skipped 1 previous similar message [ 545.668713] Lustre: server umount lustre-MDT0000 complete [ 548.283268] LustreError: 34840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830724 with bad export cookie 15835331093276699560 [ 548.287919] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.300709] LustreError: 34840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 548.497137] Lustre: server umount lustre-MDT0001 complete [ 555.894976] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 557.593747] Key type lgssc unregistered [ 557.810959] LNet: 38389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 557.817291] LNetError: 38389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 557.837602] LNet: Removed LNI 192.168.203.159@tcp [ 558.419135] Key type .llcrypt unregistered [ 558.421043] Key type ._llcrypt unregistered [ 572.586491] Key type ._llcrypt registered [ 572.589295] Key type .llcrypt registered [ 572.673196] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 573.228328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 573.294038] alg: No test for adler32 (adler32-zlib) [ 574.274718] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 574.442744] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 576.096306] Key type lgssc registered [ 576.984381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 581.601812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 581.623303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.943896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.576433] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 591.546827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.767049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 593.883920] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 597.312539] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 600.078477] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 602.908146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 603.060377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.711067] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 608.178421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 609.320784] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 621.541548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.543556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 621.555378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 621.568457] Lustre: Skipped 1 previous similar message [ 624.771324] Lustre: server umount lustre-OST0000 complete [ 646.117251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.136505] Lustre: Skipped 1 previous similar message [ 646.147188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 646.154015] Lustre: Skipped 2 previous similar messages [ 651.237104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 651.247571] Lustre: Skipped 1 previous similar message [ 656.361686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 656.375904] Lustre: Skipped 1 previous similar message [ 658.400144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 660.641267] Lustre: server umount lustre-MDT0000 complete [ 661.477608] LustreError: 39779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 661.495375] LustreError: 39779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 663.498794] LustreError: 39761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830839 with bad export cookie 11624951350887090064 [ 663.499837] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 663.520489] LustreError: 39761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 669.907535] Lustre: server umount lustre-MDT0001 complete [ 675.804561] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 676.970704] Key type lgssc unregistered [ 677.118586] LNet: 43489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.121637] LNetError: 43489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.136322] LNet: Removed LNI 192.168.203.159@tcp [ 677.491613] Key type .llcrypt unregistered [ 677.493285] Key type ._llcrypt unregistered [ 696.373693] Key type ._llcrypt registered [ 696.376964] Key type .llcrypt registered [ 696.463542] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 697.533406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 697.660287] alg: No test for adler32 (adler32-zlib) [ 698.802100] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 699.010803] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 700.664766] Key type lgssc registered [ 701.669086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 709.306641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 709.325898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.679318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 713.255046] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 720.866118] LustreError: 44880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.885274] LustreError: 44880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 723.686842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.001809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 726.828907] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 732.176938] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 736.357890] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 740.935331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 741.212720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 745.225501] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 750.402444] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 755.574373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 760.801823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.803630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 760.819298] Lustre: Skipped 2 previous similar messages [ 760.830082] Lustre: Skipped 2 previous similar messages [ 762.160743] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 765.923760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 765.930850] Lustre: Skipped 2 previous similar messages [ 766.503510] Lustre: server umount lustre-MDT0000 complete [ 767.263067] LustreError: 44876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 768.668026] LustreError: 47459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830944 with bad export cookie 12316381677280681499 [ 768.668450] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.677146] LustreError: 47459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 771.041319] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 771.044085] LustreError: 45995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 771.050190] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 773.089674] LustreError: 44875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.104939] LustreError: 44875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 774.871273] Lustre: server umount lustre-MDT0001 complete [ 802.273318] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 802.369275] Lustre: server umount lustre-OST0000 complete [ 814.553191] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 816.409660] Key type lgssc unregistered [ 816.574399] LNet: 48743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.580674] LNetError: 48743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.602332] LNet: Removed LNI 192.168.203.159@tcp [ 817.248641] Key type .llcrypt unregistered [ 817.250954] Key type ._llcrypt unregistered [ 834.796888] Key type ._llcrypt registered [ 834.799324] Key type .llcrypt registered [ 834.876711] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 835.486537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 835.527106] alg: No test for adler32 (adler32-zlib) [ 836.601833] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 836.791566] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 838.432124] Key type lgssc registered [ 839.642735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 846.767877] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 846.806150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 862.432437] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 867.554048] LustreError: 50122:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 894.176239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.466593] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 903.740627] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1067.477150] Lustre: server umount lustre-OST0000 complete [ 1081.178906] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1083.184735] Key type lgssc unregistered [ 1083.515698] LNet: 51804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.531911] LNetError: 51804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.547180] LNet: Removed LNI 192.168.203.159@tcp [ 1084.202795] Key type .llcrypt unregistered [ 1084.205537] Key type ._llcrypt unregistered [ 1102.925545] Key type ._llcrypt registered [ 1102.928491] Key type .llcrypt registered [ 1103.067676] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1103.942191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1104.049883] alg: No test for adler32 (adler32-zlib) [ 1105.154973] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1105.362223] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1106.992160] Key type lgssc registered [ 1107.855659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1114.507489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1114.538956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.914845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.511842] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1125.345690] LustreError: 53195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.375409] LustreError: 53195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1127.517186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1127.827519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1131.284396] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1137.738835] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1142.316683] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1148.085961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1148.355474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1153.089872] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1159.331089] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1160.653210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1162.721403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1162.734057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.754844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1165.805114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1165.811897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.815083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1167.511493] Lustre: server umount lustre-OST0000 complete [ 1173.986294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1173.996355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1174.015530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1174.018359] Lustre: Skipped 1 previous similar message [ 1176.045956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.047843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1176.076595] Lustre: Skipped 1 previous similar message [ 1177.695541] Lustre: server umount lustre-MDT0000 complete [ 1180.984881] LustreError: 53175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831357 with bad export cookie 2335884266484936205 [ 1180.987587] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1180.992511] LustreError: 53175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1181.283822] Lustre: server umount lustre-MDT0001 complete [ 1190.348438] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1193.165971] Key type lgssc unregistered [ 1193.422502] LNet: 56726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.428362] LNetError: 56726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1193.451288] LNet: Removed LNI 192.168.203.159@tcp [ 1194.315893] Key type .llcrypt unregistered [ 1194.321771] Key type ._llcrypt unregistered [ 1219.130409] Key type ._llcrypt registered [ 1219.134208] Key type .llcrypt registered [ 1219.221357] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1220.501398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1220.575648] alg: No test for adler32 (adler32-zlib) [ 1221.609941] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1221.886396] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1223.552189] Key type lgssc registered [ 1224.574067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1231.253907] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1231.274567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1246.817367] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1251.942095] LustreError: 58112:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 1278.625043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1283.973213] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1291.014505] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1297.470748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1297.880382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.953894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1301.158449] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1303.011050] LustreError: 58138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.058040] LustreError: 58138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1308.135487] LustreError: 58137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1310.566423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1310.879066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1315.781531] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1323.137353] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1330.358996] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1333.729609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.736924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1333.747806] Lustre: Skipped 1 previous similar message [ 1338.859973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1338.870885] Lustre: Skipped 2 previous similar messages [ 1343.975552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1343.987820] Lustre: Skipped 1 previous similar message [ 1347.040125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1347.178226] Lustre: server umount lustre-OST0000 complete [ 1369.569256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.590849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1369.600899] Lustre: Skipped 1 previous similar message [ 1374.689386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1374.698426] Lustre: Skipped 1 previous similar message [ 1375.714487] LustreError: 59153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.734305] LustreError: 59153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1375.790633] Lustre: server umount lustre-MDT0000 complete [ 1379.792057] LustreError: 58119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831555 with bad export cookie 9219965443833195451 [ 1379.801282] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1379.801709] LustreError: 58119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1394.145424] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1394.408764] Lustre: server umount lustre-MDT0001 complete [ 1403.945164] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1406.083904] Key type lgssc unregistered [ 1406.324841] LNet: 61861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.336937] LNetError: 61861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.355610] LNet: Removed LNI 192.168.203.159@tcp [ 1406.995148] Key type .llcrypt unregistered [ 1406.998544] Key type ._llcrypt unregistered [ 1430.267897] Key type ._llcrypt registered [ 1430.269697] Key type .llcrypt registered [ 1430.408958] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1431.175894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1431.389284] alg: No test for adler32 (adler32-zlib) [ 1432.434794] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1432.621445] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1434.256156] Key type lgssc registered [ 1435.098177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1442.347255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1442.371234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.744304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.087096] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1453.028832] LustreError: 63276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.044682] LustreError: 63276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1454.779928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1455.217143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1459.368424] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1466.055317] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1471.089965] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1477.092721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1477.528300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.098640] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1489.021771] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1489.851142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1510.371928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1510.379642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.397829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1510.419952] Lustre: Skipped 1 previous similar message [ 1510.437241] Lustre: Skipped 1 previous similar message [ 1514.596475] Lustre: server umount lustre-OST0000 complete [ 1520.610328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.614948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.620686] Lustre: Skipped 1 previous similar message [ 1520.654465] Lustre: Skipped 1 previous similar message [ 1525.274404] Lustre: server umount lustre-MDT0000 complete [ 1525.730272] LustreError: 63275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.755300] LustreError: 63275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1528.872950] LustreError: 63257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831704 with bad export cookie 17153924131084839422 [ 1528.883181] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1528.885825] LustreError: 63257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1529.170749] Lustre: server umount lustre-MDT0001 complete [ 1538.279132] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1540.352729] Key type lgssc unregistered [ 1540.635825] LNet: 66830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.642243] LNetError: 66830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.668591] LNet: Removed LNI 192.168.203.159@tcp [ 1541.459735] Key type .llcrypt unregistered [ 1541.465414] Key type ._llcrypt unregistered [ 1569.223721] Key type ._llcrypt registered [ 1569.225471] Key type .llcrypt registered [ 1569.353775] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1570.523746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1570.545290] alg: No test for adler32 (adler32-zlib) [ 1571.500853] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1571.739261] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1573.352648] Key type lgssc registered [ 1574.196810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1583.022270] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 07:55:58 (1787831758) [ 1589.901056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1589.929278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.506520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1596.668657] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1601.015176] LustreError: 68536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.045828] LustreError: 68536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1604.678538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.165738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1609.246321] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1616.126798] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1621.998875] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1628.290662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1628.623886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1635.103160] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1636.873676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1642.227839] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1647.648357] Lustre: Failing over lustre-MDT0000 [ 1647.970972] Lustre: server umount lustre-MDT0000 complete [ 1648.964446] LustreError: 68532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1651.171395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1651.191598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.205551] LustreError: 68535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.197308] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.215257] Lustre: Skipped 1 previous similar message [ 1654.070500] LustreError: 68532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1654.106291] LustreError: 68532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1657.146783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.351428] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.482892] Lustre: *** cfs_fail_loc=135, val=0*** [ 1657.590388] LustreError: 71621:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1657.601453] LustreError: 71621:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.159@tcp: cfg command failed: rc = -2 [ 1657.612559] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1657.612559] [ 1657.619285] LustreError: MGC192.168.203.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1657.632286] LustreError: 71609:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1657.641368] LustreError: 71609:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1657.649915] LustreError: 71609:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1657.657628] LustreError: 71609:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1657.681038] Lustre: server umount lustre-MDT0000 complete [ 1657.689481] LustreError: 71609:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1659.168593] LustreError: 69776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1659.187835] LustreError: 69776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1664.365875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.457330] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1664.467610] LustreError: 68529:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1664.713966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.766517] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1669.311303] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1669.413292] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1670.135194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1670.185926] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1670.237751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1680.364971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.374225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1680.388717] Lustre: Skipped 1 previous similar message [ 1683.639473] Lustre: server umount lustre-OST0000 complete [ 1690.599602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.599922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1690.603042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1690.603050] Lustre: Skipped 1 previous similar message [ 1690.626350] Lustre: Skipped 1 previous similar message [ 1693.636343] Lustre: server umount lustre-MDT0000 complete [ 1695.719526] LustreError: 69776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1695.741238] LustreError: 69776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 1697.433695] LustreError: 71995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831873 with bad export cookie 10519517565905725212 [ 1697.446441] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1698.027375] Lustre: server umount lustre-MDT0001 complete [ 1709.525509] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1712.331837] Key type lgssc unregistered [ 1712.682584] LNet: 73404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.693053] LNetError: 73404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.735898] LNet: Removed LNI 192.168.203.159@tcp [ 1713.678643] Key type .llcrypt unregistered [ 1713.682497] Key type ._llcrypt unregistered [ 1736.720599] Key type ._llcrypt registered [ 1736.726777] Key type .llcrypt registered [ 1736.834569] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1737.777802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1737.821810] alg: No test for adler32 (adler32-zlib) [ 1738.874259] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1739.092332] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1740.752247] Key type lgssc registered [ 1741.782821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1748.474338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1748.489188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1749.919851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.487360] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1760.233296] LustreError: 74820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1760.255255] LustreError: 74820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1762.577630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.956987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1767.336801] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1775.079764] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1780.757450] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1786.750190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1787.003756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.614901] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1795.196826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1800.513486] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1806.659631] Lustre: Failing over lustre-MDT0000 [ 1806.991732] Lustre: server umount lustre-MDT0000 complete [ 1808.208487] LustreError: 74815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1808.867273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1808.874927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.927971] LustreError: 75547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.931712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.945947] LustreError: 75547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1810.971452] Lustre: Skipped 1 previous similar message [ 1813.303318] LustreError: 74816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1813.328876] LustreError: 74816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1815.642691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.774180] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1816.000381] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1816.040898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1816.056629] Lustre: Skipped 2 previous similar messages [ 1816.162501] LustreError: 77906:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1816.170897] LustreError: 77906:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.159@tcp: cfg command failed: rc = -9 [ 1816.184582] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1816.184582] [ 1816.193601] LustreError: MGC192.168.203.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1816.210603] LustreError: 77894:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1816.217558] LustreError: 77894:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 1816.227743] LustreError: 77894:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1816.238871] LustreError: 77894:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1816.271800] Lustre: server umount lustre-MDT0000 complete [ 1816.275158] LustreError: 77894:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1818.399195] LustreError: 76287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1822.533706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.733643] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1822.912723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.974239] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1823.519389] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1827.132689] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1828.339565] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1828.370845] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1828.431794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1834.465682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1834.478413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.494940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1838.563775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1838.572481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.543273] Lustre: server umount lustre-OST0000 complete [ 1848.803096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.803322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1848.805921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1848.805929] Lustre: Skipped 1 previous similar message [ 1848.814490] Lustre: Skipped 1 previous similar message [ 1850.249924] Lustre: server umount lustre-MDT0000 complete [ 1853.150761] LustreError: 74798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832029 with bad export cookie 46658750932416069 [ 1853.163327] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1853.450484] Lustre: server umount lustre-MDT0001 complete [ 1861.980497] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1864.249378] Key type lgssc unregistered [ 1864.570938] LNet: 79689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.587217] LNetError: 79689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.611036] LNet: Removed LNI 192.168.203.159@tcp [ 1865.322393] Key type .llcrypt unregistered [ 1865.326636] Key type ._llcrypt unregistered [ 1877.260475] Key type ._llcrypt registered [ 1877.262636] Key type .llcrypt registered [ 1877.395697] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1877.649512] Key type .llcrypt unregistered [ 1877.655909] Key type ._llcrypt unregistered [ 1900.809760] Key type ._llcrypt registered [ 1900.811866] Key type .llcrypt registered [ 1900.917744] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1901.849560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1902.008480] alg: No test for adler32 (adler32-zlib) [ 1903.044790] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1903.273712] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1904.949591] Key type lgssc registered [ 1906.072736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1912.666730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1912.687054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.079938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.232384] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1924.583201] LustreError: 81607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1924.616880] LustreError: 81607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1926.122981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.574020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1930.339480] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1936.860176] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1942.102192] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1948.176251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1948.460639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.885798] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1959.731118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 1959.895674] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1970.153930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.157432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1970.183894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1975.027795] Lustre: server umount lustre-OST0000 complete [ 1980.386352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.398056] Lustre: Skipped 1 previous similar message [ 1980.408824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1980.412035] Lustre: Skipped 2 previous similar messages [ 1984.247366] Lustre: server umount lustre-MDT0000 complete [ 1985.510537] LustreError: 81607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1985.530938] LustreError: 81607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1987.456024] LustreError: 81587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832163 with bad export cookie 8285841793082638291 [ 1987.460899] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1987.467964] LustreError: 81587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1987.779309] Lustre: server umount lustre-MDT0001 complete [ 1997.799275] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2000.468561] Key type lgssc unregistered [ 2000.836469] LNet: 85142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.842868] LNetError: 85142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.863670] LNet: Removed LNI 192.168.203.159@tcp [ 2001.776402] Key type .llcrypt unregistered [ 2001.782068] Key type ._llcrypt unregistered [ 2022.956322] Key type ._llcrypt registered [ 2022.958481] Key type .llcrypt registered [ 2023.053823] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2023.878659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2023.968842] alg: No test for adler32 (adler32-zlib) [ 2025.030165] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 2025.275792] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2026.947596] Key type lgssc registered [ 2027.967902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2034.653698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2034.671124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.100212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.564355] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2046.447205] LustreError: 86555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.471380] LustreError: 86555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2046.799836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.175317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2050.436559] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2056.188230] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2060.848576] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2066.309971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2066.613529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.627780] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2077.850969] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2080.946921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2088.420895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2088.434991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.466030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2091.491683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.499919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2091.513900] Lustre: Skipped 1 previous similar message [ 2094.431619] Lustre: server umount lustre-OST0000 complete [ 2098.661041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2098.669518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.682939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2101.729692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2101.740114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.410261] Lustre: server umount lustre-MDT0000 complete [ 2106.850803] LustreError: 86555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.867951] LustreError: 86555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2108.015275] LustreError: 86537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832284 with bad export cookie 13310002999668638509 [ 2108.031542] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2108.043834] LustreError: 86537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2108.383882] Lustre: server umount lustre-MDT0001 complete [ 2117.643469] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2119.972818] Key type lgssc unregistered [ 2120.258507] LNet: 90090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.271874] LNetError: 90090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.286140] LNet: Removed LNI 192.168.203.159@tcp [ 2121.100336] Key type .llcrypt unregistered [ 2121.102736] Key type ._llcrypt unregistered [ 2143.219245] Key type ._llcrypt registered [ 2143.225630] Key type .llcrypt registered [ 2143.326566] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2144.243718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2144.273769] alg: No test for adler32 (adler32-zlib) [ 2145.338051] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 2145.578885] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2147.280170] Key type lgssc registered [ 2148.629883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2156.133869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2156.161147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.585377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2161.783085] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2167.788654] LustreError: 91521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.811736] LustreError: 91521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2169.313082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.573408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2172.591375] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2177.643275] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2182.255083] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2187.471404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2187.813166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.609216] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2199.874201] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2202.164679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2213.864923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2213.887410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.900930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.369482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2217.443319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2217.450741] Lustre: Skipped 2 previous similar messages [ 2219.020052] Lustre: server umount lustre-OST0000 complete [ 2226.147401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2226.154822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.169847] Lustre: Skipped 1 previous similar message [ 2226.176710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.684137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.550246] Lustre: server umount lustre-MDT0000 complete [ 2232.807740] LustreError: 91521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.834018] LustreError: 91521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2233.684385] LustreError: 91501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832409 with bad export cookie 12537704940916248260 [ 2233.696169] LustreError: 91501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2233.696230] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2234.096202] Lustre: server umount lustre-MDT0001 complete [ 2245.082871] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2247.802904] Key type lgssc unregistered [ 2248.065966] LNet: 95075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.072794] LNetError: 95075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.090077] LNet: Removed LNI 192.168.203.159@tcp [ 2248.859555] Key type .llcrypt unregistered [ 2248.862658] Key type ._llcrypt unregistered [ 2270.033118] Key type ._llcrypt registered [ 2270.036882] Key type .llcrypt registered [ 2270.173581] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2271.391652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.432776] alg: No test for adler32 (adler32-zlib) [ 2272.442530] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 2272.629180] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2274.304180] Key type lgssc registered [ 2275.196867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2282.031800] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2282.047508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2297.570870] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2302.690632] LustreError: 96455:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 2329.312297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.504446] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2343.953689] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2349.038344] Lustre: server umount lustre-OST0000 complete [ 2356.678596] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 08:08:51 (1787832531) [ 2363.351922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.706977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.086357] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2368.997517] LustreError: 98092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.016094] LustreError: 98092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2373.090060] LustreError: 98091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.118511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.407283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2380.071425] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2385.932180] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2391.233142] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2396.543536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.830269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2401.949915] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2408.110318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2408.846982] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2418.660592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2418.666283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2418.680334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2418.687726] Lustre: Skipped 1 previous similar message [ 2423.787546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2423.805157] Lustre: Skipped 1 previous similar message [ 2424.472055] Lustre: server umount lustre-OST0000 complete [ 2428.898089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.909658] Lustre: Skipped 1 previous similar message [ 2428.914038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2434.018488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2434.026525] Lustre: Skipped 2 previous similar messages [ 2434.446599] Lustre: server umount lustre-MDT0000 complete [ 2437.929488] LustreError: 98072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832614 with bad export cookie 8920055263752534388 [ 2437.936992] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2437.942839] LustreError: 98072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2438.268444] Lustre: server umount lustre-MDT0001 complete [ 2448.436949] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2450.962512] Key type lgssc unregistered [ 2451.199269] LNet: 101786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2451.208115] LNetError: 101786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2451.225958] LNet: Removed LNI 192.168.203.159@tcp [ 2451.973460] Key type .llcrypt unregistered [ 2451.975974] Key type ._llcrypt unregistered [ 2485.604649] Key type ._llcrypt registered [ 2485.608181] Key type .llcrypt registered [ 2485.734625] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2486.976387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2487.002851] alg: No test for adler32 (adler32-zlib) [ 2488.172195] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 2488.451397] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2490.153885] Key type lgssc registered [ 2491.271202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2498.541587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2498.568669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2499.943676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.804035] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2510.309628] LustreError: 103952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.320506] LustreError: 103952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2511.961665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.403742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2517.365683] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2525.871252] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2531.795627] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2537.481149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2537.858689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.180960] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2545.008675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2549.398612] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2567.147266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2567.164234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.188806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2567.650037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2569.270177] Lustre: server umount lustre-OST0000 complete [ 2573.798691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2573.804675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.814818] Lustre: Skipped 1 previous similar message [ 2573.819958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.827061] Lustre: Skipped 1 previous similar message [ 2577.891827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2577.892652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2583.010903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2583.022521] Lustre: Skipped 2 previous similar messages [ 2587.109164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2587.111393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.119868] Lustre: Skipped 1 previous similar message [ 2587.245067] Lustre: server umount lustre-MDT0000 complete [ 2590.467723] LustreError: 103934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832766 with bad export cookie 2168113789382073736 [ 2590.477220] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2590.486793] LustreError: 103934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2590.716846] Lustre: server umount lustre-MDT0001 complete [ 2601.711989] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2604.656985] Key type lgssc unregistered [ 2604.977669] LNet: 107509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2604.990513] LNetError: 107509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2605.019272] LNet: Removed LNI 192.168.203.159@tcp [ 2605.790465] Key type .llcrypt unregistered [ 2605.794056] Key type ._llcrypt unregistered [ 2623.379829] Key type ._llcrypt registered [ 2623.383694] Key type .llcrypt registered [ 2623.525766] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2625.173598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2625.249496] alg: No test for adler32 (adler32-zlib) [ 2626.355690] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 2626.657754] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2628.344467] Key type lgssc registered [ 2629.837618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2638.519115] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2638.569581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2654.241759] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2659.360406] LustreError: 108752:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 2686.049941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.272616] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2698.641616] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2704.914547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.187342] LustreError: MGC192.168.203.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2705.196660] LustreError: 109776:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2705.202968] LustreError: 109776:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2705.216128] LustreError: 109776:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2705.375750] Lustre: server umount lustre-MDT0000 complete [ 2705.379773] LustreError: 109776:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2726.880180] Lustre: 108088:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832886/real 1787832886] req@0000000073b7e49b x1874678568193280/t0(0) o400->MGC192.168.203.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787832902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2726.895204] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2726.907122] LustreError: 108765:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2727.096720] Lustre: server umount lustre-OST0000 complete [ 2735.170352] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2744.855637] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2778.490306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.792454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2778.850620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2778.958618] Lustre: lustre-MDT0000: new disk, initializing [ 2779.035495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.049195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2784.196179] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2795.943298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.103899] Lustre: 112567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2796.167119] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2796.171819] Lustre: Skipped 1 previous similar message [ 2796.286067] Lustre: lustre-MDT0001: new disk, initializing [ 2796.432883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2796.490725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2796.500154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2800.814870] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2809.710946] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2815.705071] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2821.563606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2821.794928] Lustre: lustre-OST0000: new disk, initializing [ 2821.801923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2821.807602] Lustre: 114536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2821.898065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2828.045630] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2831.427232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2831.445087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2831.546758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2837.197402] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2842.014802] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2842.350605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2847.015406] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2847.243882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2852.321795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2852.323665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.335909] LustreError: Skipped 1 previous similar message [ 2852.338560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2852.350971] Lustre: Skipped 1 previous similar message [ 2852.381586] Lustre: Skipped 1 previous similar message [ 2855.513406] Lustre: server umount lustre-OST0000 complete [ 2859.503316] LustreError: 108090:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005f25d3ec x1874678568232320/t0(0) o41->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2862.568313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.579590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.593853] Lustre: Skipped 1 previous similar message [ 2865.975978] Lustre: server umount lustre-MDT0000 complete [ 2869.874149] LustreError: 112559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833045 with bad export cookie 799026487064354346 [ 2869.889992] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2870.246034] Lustre: server umount lustre-MDT0001 complete [ 2878.450369] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 08:17:34 (1787833054) [ 2881.433561] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2902.365625] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2912.676058] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2946.862383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2947.178239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2947.232539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2947.357302] Lustre: lustre-MDT0000: new disk, initializing [ 2947.491706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.525346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2952.231423] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2962.743857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.894784] Lustre: 119436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2962.970942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2962.975227] Lustre: Skipped 1 previous similar message [ 2963.104987] Lustre: lustre-MDT0001: new disk, initializing [ 2963.316664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2963.368448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2963.388203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2968.950121] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2979.081415] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2985.302804] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2993.440097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2993.882391] Lustre: lustre-OST0000: new disk, initializing [ 2993.888192] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2993.905944] Lustre: 121406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2994.083425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.053532] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3003.490252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3003.501855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3003.629221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3012.345916] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3016.751179] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3016.885859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3020.908184] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3021.140951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3024.354179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3024.355792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3024.368527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3024.368535] Lustre: Skipped 1 previous similar message [ 3024.396824] Lustre: Skipped 1 previous similar message [ 3029.485094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3029.497312] Lustre: Skipped 2 previous similar messages [ 3029.769766] Lustre: server umount lustre-OST0000 complete [ 3034.599249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.604222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3034.624032] Lustre: Skipped 1 previous similar message [ 3034.653565] Lustre: Skipped 1 previous similar message [ 3039.715906] LustreError: 120805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.741894] LustreError: 120805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3039.983519] Lustre: server umount lustre-MDT0000 complete [ 3044.314103] LustreError: 119431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833220 with bad export cookie 799026487064355228 [ 3044.329524] LustreError: 119431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3044.333516] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3044.665160] Lustre: server umount lustre-MDT0001 complete [ 3053.957097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.715970] LustreError: 123623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.817937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.206709] LustreError: 123624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.455956] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3064.363311] LustreError: 123623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.410422] LustreError: 123624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.767724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.058716] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3083.039726] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3089.245545] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3096.206224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3103.216431] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3110.712350] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3120.665096] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3125.740214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3125.749399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.753034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3125.761009] Lustre: Skipped 1 previous similar message [ 3132.040550] Lustre: server umount lustre-OST0000 complete [ 3136.993552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3137.001111] LustreError: Skipped 1 previous similar message [ 3137.008981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.209588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3146.231034] Lustre: Skipped 7 previous similar messages [ 3151.329779] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3151.595731] Lustre: server umount lustre-MDT0000 complete [ 3155.361117] LustreError: 123604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833331 with bad export cookie 799026487064356110 [ 3155.363949] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.376880] LustreError: 123604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3155.774081] Lustre: server umount lustre-MDT0001 complete [ 3166.419868] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3169.084319] Key type lgssc unregistered [ 3169.342382] LNet: 127259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3169.351908] LNetError: 127259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3169.366158] LNet: Removed LNI 192.168.203.159@tcp [ 3170.397285] Key type .llcrypt unregistered [ 3170.399817] Key type ._llcrypt unregistered [ 3196.909435] Key type ._llcrypt registered [ 3196.912492] Key type .llcrypt registered [ 3197.048630] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 3214.639606] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3216.011331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3216.134024] alg: No test for adler32 (adler32-zlib) [ 3217.249637] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 3217.574474] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3219.331712] Key type lgssc registered [ 3220.557730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3256.364818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3256.458283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.829457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3257.851916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3257.933342] Lustre: lustre-MDT0000: new disk, initializing [ 3258.036549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3258.062586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3264.278552] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3275.337132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.484085] Lustre: 130421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3275.565351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3275.571463] Lustre: Skipped 1 previous similar message [ 3275.658724] Lustre: lustre-MDT0001: new disk, initializing [ 3275.787574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3275.828710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3275.850943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3280.550757] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3290.705150] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3296.954813] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3304.455805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3304.807245] Lustre: lustre-OST0000: new disk, initializing [ 3304.810531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3304.819325] Lustre: 132389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3304.952436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3312.333074] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3313.203718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3313.218916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3313.381372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3322.637748] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3327.661678] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3327.924441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3333.442467] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3333.709786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3339.236764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3339.243530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3339.268162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3339.276985] Lustre: Skipped 1 previous similar message [ 3341.873473] Lustre: server umount lustre-OST0000 complete [ 3347.425321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3347.431113] LustreError: Skipped 1 previous similar message [ 3347.438585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3347.454933] Lustre: Skipped 1 previous similar message [ 3347.466434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3349.474760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.475827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3349.508796] Lustre: Skipped 1 previous similar message [ 3352.520802] Lustre: server umount lustre-MDT0000 complete [ 3357.307415] LustreError: 130411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833533 with bad export cookie 17423255899003711936 [ 3357.311804] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3357.315705] LustreError: 130411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3357.554618] Lustre: server umount lustre-MDT0001 complete [ 3366.188169] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 08:25:41 (1787833541) [ 3373.318389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.701939] LustreError: 134802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3373.710860] LustreError: 134802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3373.743600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3377.781310] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3379.183743] LustreError: 134803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3383.305979] LustreError: 134802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3388.396647] LustreError: 134803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3389.922381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.418763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3395.752226] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3404.732592] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3411.615482] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3415.332550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.338773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3415.345788] Lustre: Skipped 1 previous similar message [ 3415.363057] Lustre: Skipped 1 previous similar message [ 3420.644848] LustreError: 134798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.667385] LustreError: 134798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3420.820210] Lustre: server umount lustre-MDT0000 complete [ 3424.959989] LustreError: 134784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833601 with bad export cookie 17423255899003712818 [ 3424.974508] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3424.977130] LustreError: 134784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3425.626118] Lustre: server umount lustre-MDT0001 complete [ 3435.006863] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 08:26:50 (1787833610) [ 3442.451467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3458.080585] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3463.200652] LustreError: 137218:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 3488.929224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.221140] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3505.491653] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3508.213897] Lustre: server umount lustre-OST0000 complete [ 3517.778253] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 08:28:13 (1787833693) [ 3524.924187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.522245] LustreError: 138674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.595494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3530.655995] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3540.869925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.558665] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3555.172440] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3561.943062] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3570.199167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3570.765071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.773993] Lustre: Skipped 1 previous similar message [ 3579.227558] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3587.233404] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3605.475786] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3605.477915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3605.507644] Lustre: Skipped 2 previous similar messages [ 3605.526093] Lustre: Skipped 2 previous similar messages [ 3610.825521] Lustre: server umount lustre-MDT0000 complete [ 3614.994278] LustreError: 138654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833791 with bad export cookie 17423255899003713483 [ 3614.995621] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3615.006402] LustreError: 138654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3615.462221] Lustre: server umount lustre-MDT0001 complete [ 3629.957992] Lustre: server umount lustre-OST0000 complete [ 3639.924452] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 08:30:15 (1787833815) [ 3648.172987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.790742] LustreError: 142352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3648.820018] LustreError: 142352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3648.894620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3654.985610] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3664.469747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3670.091712] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3678.051593] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3684.158563] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3692.027505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3693.455441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3699.195804] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3706.588763] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3710.855677] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3711.105178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3716.030316] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3716.235615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3719.139080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3719.147550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.170687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3719.182749] Lustre: Skipped 3 previous similar messages [ 3724.488334] Lustre: server umount lustre-OST0000 complete [ 3734.840552] Lustre: server umount lustre-MDT0000 complete [ 3735.524789] LustreError: 143504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3735.545760] LustreError: 143504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3738.551527] LustreError: 142333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833914 with bad export cookie 17423255899003714554 [ 3738.555223] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3738.576697] LustreError: 142333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3739.027696] Lustre: server umount lustre-MDT0001 complete [ 3745.822956] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 08:32:01 (1787833921) [ 3751.822369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3767.392950] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3772.512520] LustreError: 146512:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 3798.176350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3798.180439] Lustre: Skipped 2 previous similar messages [ 3804.162388] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3813.614977] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3820.578898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3821.053373] LustreError: 146537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3821.219957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3825.557948] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3834.519334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3839.817765] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3847.869796] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3854.223858] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3859.432643] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3859.644569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3864.045588] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3864.251899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3866.084288] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3866.088335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3866.098219] Lustre: Skipped 5 previous similar messages [ 3866.107768] Lustre: Skipped 5 previous similar messages [ 3872.461550] Lustre: server umount lustre-MDT0000 complete [ 3875.958773] LustreError: 146518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834052 with bad export cookie 17423255899003715576 [ 3875.961048] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3875.977880] LustreError: 146518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3876.370417] Lustre: server umount lustre-MDT0001 complete [ 3890.779945] Lustre: server umount lustre-OST0000 complete [ 3899.651648] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 08:34:35 (1787834075) [ 3906.577623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3922.145708] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3927.265835] LustreError: 150708:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 3960.170896] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3967.825649] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3974.781052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.303383] LustreError: 150733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.319751] LustreError: 150733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3975.429456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3979.391496] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3989.055719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3994.913737] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4002.189741] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4007.510503] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4013.618398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4013.888972] Lustre: lustre-OST0001: new disk, initializing [ 4013.896307] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4013.904540] Lustre: 153578:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4020.577453] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4021.346429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4021.368436] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4021.476570] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4029.801199] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4034.455606] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4034.700737] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4039.634943] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4039.907342] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4042.213106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.247933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4042.258475] Lustre: Skipped 4 previous similar messages [ 4048.043725] Lustre: server umount lustre-OST0000 complete [ 4052.449652] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4052.457024] LustreError: Skipped 2 previous similar messages [ 4058.697983] Lustre: server umount lustre-OST0001 complete [ 4068.778467] Lustre: server umount lustre-MDT0000 complete [ 4073.266626] LustreError: 150715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834249 with bad export cookie 17423255899003716528 [ 4073.272456] LustreError: 150715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4073.272648] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4073.462175] Lustre: server umount lustre-MDT0001 complete [ 4097.127724] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 08:37:52 (1787834272) [ 4099.242438] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4100.787535] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4102.182761] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 08:37:58 (1787834278) [ 4109.733050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4110.216614] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4110.261564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4110.696445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4110.710829] Lustre: Skipped 6 previous similar messages [ 4116.416484] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4124.715329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4124.859507] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4124.923234] Lustre: 157141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4130.555961] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4137.976483] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4143.629560] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4150.021162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4150.280604] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4156.843082] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4159.140966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4163.876408] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4168.098698] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4168.347512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4172.748251] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4172.939109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4179.431808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4179.445688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.465135] Lustre: Skipped 5 previous similar messages [ 4179.470893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4179.476240] Lustre: Skipped 10 previous similar messages [ 4189.152203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4189.505234] Lustre: server umount lustre-OST0000 complete [ 4211.881552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4218.144461] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4225.442143] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4230.703342] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4235.823953] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4235.838621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4235.841335] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4235.876952] Lustre: Skipped 1 previous similar message [ 4236.217914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 4240.569751] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4240.739458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4246.016220] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f614627e800.ost_server_uuid 50 [ 4247.596746] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f614627e800.ost_server_uuid in FULL state after 0 sec [ 4247.747160] LustreError: 157146:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.59@tcp) returned error from blocking AST (req@00000000cb523f3f x1874679188163584 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000023d66bfd/0xf1cbd6ed82d88d0a lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.59@tcp remote: 0xb2de273add029c54 expref: 5 pid: 157148 timeout: 4347 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4247.775885] LustreError: lustre-MDT0000: A client on nid 192.168.203.59@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4247.783614] LustreError: 157136:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.59@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000023d66bfd/0xf1cbd6ed82d88d0a lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.59@tcp remote: 0xb2de273add029c54 expref: 6 pid: 157148 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4252.497935] Lustre: server umount lustre-OST0000 complete [ 4266.559377] Lustre: server umount lustre-MDT0000 complete [ 4269.944028] LustreError: 157131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834446 with bad export cookie 17423255899003717711 [ 4269.944328] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4269.955087] LustreError: 157131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4284.385165] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4284.569814] Lustre: server umount lustre-MDT0001 complete [ 4295.062252] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4297.538790] Key type lgssc unregistered [ 4297.807735] LNet: 163293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4297.816124] LNetError: 163293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4297.832459] LNet: Removed LNI 192.168.203.159@tcp [ 4298.647152] Key type .llcrypt unregistered [ 4298.651417] Key type ._llcrypt unregistered [ 4318.687371] Key type ._llcrypt registered [ 4318.690271] Key type .llcrypt registered [ 4318.802932] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4319.802285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4319.896594] alg: No test for adler32 (adler32-zlib) [ 4320.954368] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 4321.236089] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4322.912154] Key type lgssc registered [ 4323.859409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4330.909595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4330.918763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.352370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4336.709268] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4342.755165] LustreError: 164685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4342.790845] LustreError: 164685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4347.130734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4347.613211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4353.411840] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4361.480912] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4366.968363] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4373.497995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4373.961187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4377.053735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4380.105491] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4387.418384] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4396.892215] Lustre: Failing over lustre-MDT0000 [ 4397.236554] Lustre: server umount lustre-MDT0000 complete [ 4397.538162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.538721] LustreError: 164682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.552521] Lustre: Skipped 2 previous similar messages [ 4397.571371] LustreError: 164682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4402.665281] LustreError: 164680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.703700] LustreError: 164680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4407.779273] LustreError: 164681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.828212] LustreError: 164681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4409.371887] LustreError: 166529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.59@tcp arrived at 1787834585 with bad export cookie 16740947607410842238 [ 4409.401059] LustreError: 166529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4412.897613] Lustre: 163997:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834573/real 1787834573] req@00000000c5b04376 x1874680345552256/t0(0) o400->MGC192.168.203.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787834589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4412.904673] LustreError: 165413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.941824] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4412.985746] LustreError: 165413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4416.976304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4423.148747] LustreError: 165413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4423.148931] LustreError: 163995:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006ee0f00c x1874680345557504/t0(0) o250->MGC192.168.203.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4423.176569] LustreError: 165413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4423.504767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.572076] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4427.771056] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4428.777296] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4428.802799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4430.054323] Lustre: lustre-MDT0000: Denying connection for new client ded38a3f-bc01-4130-a9aa-0b9ef4547017 (at 192.168.203.59@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4438.499384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4438.506453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4438.528058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4444.133338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4444.145516] Lustre: Skipped 3 previous similar messages [ 4444.645476] Lustre: server umount lustre-OST0000 complete [ 4448.013937] LustreError: 168418:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4448.023620] Lustre: 167767:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4448.037825] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4448.064726] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4448.071365] LustreError: 167767:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000074028d97 x1874680345571072/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4448.079812] LustreError: 167767:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4448.084524] LustreError: 167767:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4448.093536] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 4448.452540] Lustre: server umount lustre-MDT0000 complete [ 4449.252537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.267689] Lustre: Skipped 1 previous similar message [ 4449.273281] LustreError: 166167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.470177] LustreError: 166529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834627 with bad export cookie 16740947607410842343 [ 4451.473372] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4451.481774] LustreError: 166529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4457.964098] Lustre: server umount lustre-MDT0001 complete [ 4468.673568] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4471.574416] Key type lgssc unregistered [ 4471.819568] LNet: 169173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4471.829411] LNetError: 169173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4471.854121] LNet: Removed LNI 192.168.203.159@tcp [ 4472.738131] Key type .llcrypt unregistered [ 4472.740803] Key type ._llcrypt unregistered [ 4497.130949] Key type ._llcrypt registered [ 4497.132933] Key type .llcrypt registered [ 4497.218489] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4498.323096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4498.385223] alg: No test for adler32 (adler32-zlib) [ 4499.463625] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 4499.661741] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4501.352176] Key type lgssc registered [ 4502.664710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4510.452811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4510.465518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4511.828958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4515.942692] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4520.930812] LustreError: 170588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.948533] LustreError: 170588:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4523.764697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.075708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4528.578409] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4536.261456] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4541.925426] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4548.368964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4548.826488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4554.531539] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4556.008889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4561.046397] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4569.605560] Lustre: server umount lustre-OST0000 complete [ 4575.203111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4575.208505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4575.225079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4576.741811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4576.743294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.860980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4581.869610] Lustre: Skipped 2 previous similar messages [ 4582.974062] Lustre: server umount lustre-MDT0000 complete [ 4586.016251] Lustre: 169897:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834746/real 1787834746] req@00000000a257dbbf x1874680532198528/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787834762 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4586.033591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.044238] LustreError: 170587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.068373] LustreError: 170587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4586.616332] LustreError: 170567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834762 with bad export cookie 3495791732540876889 [ 4586.637652] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4587.114312] Lustre: server umount lustre-MDT0001 complete [ 4596.622676] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4598.847748] Key type lgssc unregistered [ 4599.043618] LNet: 174123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4599.052572] LNetError: 174123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4599.072562] LNet: Removed LNI 192.168.203.159@tcp [ 4599.791345] Key type .llcrypt unregistered [ 4599.793451] Key type ._llcrypt unregistered [ 4630.852510] Key type ._llcrypt registered [ 4630.858619] Key type .llcrypt registered [ 4631.025198] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4631.995239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4632.045582] alg: No test for adler32 (adler32-zlib) [ 4633.104824] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 4633.459376] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4635.129525] Key type lgssc registered [ 4636.159213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4643.928603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4643.943922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4645.366734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.612885] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4655.585955] LustreError: 175824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.621381] LustreError: 175824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4658.107517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4658.532956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4663.374566] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4670.333700] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4675.404343] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4681.849670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4682.143262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4687.850648] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4690.355277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4694.119444] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4703.583980] Lustre: server umount lustre-OST0000 complete [ 4709.865953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4709.874046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4709.882883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4710.884827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4710.888426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4710.909482] Lustre: Skipped 1 previous similar message [ 4716.003661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4716.011249] Lustre: Skipped 1 previous similar message [ 4717.138752] Lustre: server umount lustre-MDT0000 complete [ 4720.096371] Lustre: 175136:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834880/real 1787834880] req@00000000abe75dc3 x1874680672708352/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787834896 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4720.118578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.135729] LustreError: 175824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4720.154259] LustreError: 175824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4720.636708] LustreError: 175804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834896 with bad export cookie 3697232757181032076 [ 4720.642901] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4720.650310] LustreError: 175804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4721.049315] Lustre: server umount lustre-MDT0001 complete [ 4731.327723] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4734.174253] Key type lgssc unregistered [ 4734.523377] LNet: 179361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4734.531761] LNetError: 179361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4734.569423] LNet: Removed LNI 192.168.203.159@tcp [ 4735.445824] Key type .llcrypt unregistered [ 4735.447900] Key type ._llcrypt unregistered [ 4758.776556] Key type ._llcrypt registered [ 4758.780489] Key type .llcrypt registered [ 4758.886577] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4760.102749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4760.166784] alg: No test for adler32 (adler32-zlib) [ 4761.164802] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 4761.389059] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4763.064190] Key type lgssc registered [ 4764.335806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4773.952897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4773.966657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4775.370495] Lustre: *** cfs_fail_loc=135, val=0*** [ 4775.456506] LustreError: 180887:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 4775.468182] LustreError: 180887:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.159@tcp: cfg command failed: rc = -2 [ 4775.474989] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4775.474989] [ 4775.500834] LustreError: MGC192.168.203.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4775.522245] LustreError: 180847:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4775.533516] LustreError: 180847:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4775.540293] LustreError: 180847:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 4775.548800] LustreError: 180847:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 4775.659488] Lustre: server umount lustre-MDT0000 complete [ 4775.665145] LustreError: 180847:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4781.953718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4782.399498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4786.981722] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4792.802327] LustreError: 181258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4792.831396] LustreError: 181258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4795.242384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.675514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4800.302793] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4808.105269] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4813.549652] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4815.545465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4815.547574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4815.562391] Lustre: Skipped 1 previous similar message [ 4815.582591] Lustre: Skipped 1 previous similar message [ 4820.713423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4820.722338] Lustre: Skipped 1 previous similar message [ 4821.898547] Lustre: server umount lustre-MDT0000 complete [ 4825.311639] LustreError: 181240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835001 with bad export cookie 16913019107322417150 [ 4825.324096] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4825.327789] LustreError: 181240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4825.676423] Lustre: server umount lustre-MDT0001 complete [ 4835.707038] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4838.247933] Key type lgssc unregistered [ 4838.530657] LNet: 183581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4838.542104] LNetError: 183581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4838.559245] LNet: Removed LNI 192.168.203.159@tcp [ 4839.454703] Key type .llcrypt unregistered [ 4839.458630] Key type ._llcrypt unregistered [ 4858.694595] Key type ._llcrypt registered [ 4858.696956] Key type .llcrypt registered [ 4858.842660] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4859.195426] Key type .llcrypt unregistered [ 4859.203881] Key type ._llcrypt unregistered [ 4875.559748] Key type ._llcrypt registered [ 4875.565884] Key type .llcrypt registered [ 4875.697733] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4876.559303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4876.716126] alg: No test for adler32 (adler32-zlib) [ 4877.774797] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 4877.959495] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4879.704141] Key type lgssc registered [ 4881.048261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4889.298587] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4889.327183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4904.928419] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4910.049410] LustreError: 185717:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 4936.736177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4942.766852] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4949.386588] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4955.877152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.537556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.624431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4960.969902] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4961.765969] LustreError: 185742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4961.798494] LustreError: 185742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4966.889434] LustreError: 185743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4969.053507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4969.476278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4973.962354] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4981.057555] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4986.625399] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4989.362796] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5000.166632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5000.177257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5000.200770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.218851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5002.224044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.259673] Lustre: Skipped 1 previous similar message [ 5005.567637] Lustre: server umount lustre-OST0000 complete [ 5010.402958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5010.417654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5010.429622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5012.458879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.459077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5012.481710] Lustre: Skipped 1 previous similar message [ 5015.736276] Lustre: server umount lustre-MDT0000 complete [ 5017.573175] LustreError: 186753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5017.601292] LustreError: 186753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5019.212626] LustreError: 185725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835195 with bad export cookie 10835586766763529905 [ 5019.219444] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5019.229448] LustreError: 185725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5019.551252] Lustre: server umount lustre-MDT0001 complete [ 5030.754383] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5033.886946] Key type lgssc unregistered [ 5034.241192] LNet: 189683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5034.249431] LNetError: 189683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5034.273564] LNet: Removed LNI 192.168.203.159@tcp [ 5035.098840] Key type .llcrypt unregistered [ 5035.101248] Key type ._llcrypt unregistered [ 5058.665603] Key type ._llcrypt registered [ 5058.667841] Key type .llcrypt registered [ 5058.798579] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5059.587982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5059.616230] alg: No test for adler32 (adler32-zlib) [ 5060.731546] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 5060.988992] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5062.672225] Key type lgssc registered [ 5063.922824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5071.489525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5071.501236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5072.935775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5077.363904] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5083.105898] LustreError: 191102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5083.132066] LustreError: 191102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5085.572343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5086.065657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5090.667711] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5098.007113] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5103.543607] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5110.531883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5110.921947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5117.801945] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5118.132621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5125.615265] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5131.089696] Lustre: Failing over lustre-MDT0000 [ 5131.483185] Lustre: server umount lustre-MDT0000 complete [ 5132.256949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5132.267821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.285406] LustreError: 191103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5133.625068] LustreError: 191099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5133.802737] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5138.729808] LustreError: 191098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5138.757342] LustreError: 191098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5143.846841] LustreError: 191097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5143.864411] LustreError: 191097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5149.156976] Lustre: 190413:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835309/real 1787835309] req@0000000053c0e1ad x1874681121500032/t0(0) o400->MGC192.168.203.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787835325 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5149.194687] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5150.632750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5154.081790] LustreError: 191099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5154.095714] LustreError: 191099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5159.394926] LustreError: 190412:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000ace397d x1874681121504768/t0(0) o250->MGC192.168.203.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5159.759532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5159.806667] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5163.683411] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5164.323109] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5165.039096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5165.063872] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5165.103192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5168.125142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5169.975471] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5180.897232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5180.902481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5180.910694] Lustre: Skipped 1 previous similar message [ 5180.916204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5183.970184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5183.979942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5183.996031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5185.535346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5186.750834] Lustre: server umount lustre-OST0000 complete [ 5190.626627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.632667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5190.658094] Lustre: Skipped 1 previous similar message [ 5190.679441] Lustre: Skipped 1 previous similar message [ 5195.747776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5195.753613] Lustre: Skipped 1 previous similar message [ 5196.801390] Lustre: server umount lustre-MDT0000 complete [ 5199.980256] LustreError: 191837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835376 with bad export cookie 7324309257558557992 [ 5199.989842] LustreError: 191837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5199.989857] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5200.245917] Lustre: server umount lustre-MDT0001 complete [ 5209.550280] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5212.349688] Key type lgssc unregistered [ 5212.670568] LNet: 195841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5212.678293] LNetError: 195841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5212.709308] LNet: Removed LNI 192.168.203.159@tcp [ 5213.840658] Key type .llcrypt unregistered [ 5213.844524] Key type ._llcrypt unregistered [ 5242.107117] Key type ._llcrypt registered [ 5242.108527] Key type .llcrypt registered [ 5242.255940] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5243.340383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5243.390028] alg: No test for adler32 (adler32-zlib) [ 5244.530873] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 5244.813993] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5246.477610] Key type lgssc registered [ 5247.392448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5254.836054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5254.845492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5256.394974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.530772] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5265.905807] LustreError: 197256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5265.933522] LustreError: 197256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5268.395151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5268.754862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5272.804721] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5279.932351] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5284.530235] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5290.364378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5290.729440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5296.422760] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5300.994916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5303.531347] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5307.453528] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5314.074531] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5327.402993] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5335.011435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5335.020103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5335.038567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5337.060985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5337.063612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5337.084480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5337.088208] Lustre: Skipped 1 previous similar message [ 5341.308862] Lustre: server umount lustre-OST0000 complete [ 5345.761203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5345.768173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5345.781741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5350.543312] Lustre: server umount lustre-MDT0000 complete [ 5352.423233] LustreError: 197252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5352.452647] LustreError: 197252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5354.092515] LustreError: 199845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835530 with bad export cookie 8181367106100500902 [ 5354.096416] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5354.102909] LustreError: 199845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5354.523680] Lustre: server umount lustre-MDT0001 complete [ 5365.298729] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5368.119972] Key type lgssc unregistered [ 5368.464793] LNet: 201109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5368.470405] LNetError: 201109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5369.514739] LNet: Removed LNI 192.168.203.159@tcp [ 5370.570553] Key type .llcrypt unregistered [ 5370.574836] Key type ._llcrypt unregistered [ 5388.853492] Key type ._llcrypt registered [ 5388.855967] Key type .llcrypt registered [ 5388.972193] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5390.056780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5390.156070] alg: No test for adler32 (adler32-zlib) [ 5391.133629] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 5391.462641] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5393.224161] Key type lgssc registered [ 5394.297874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5402.360883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5402.386331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.113932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5410.590608] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5414.418100] LustreError: 202378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5414.436827] LustreError: 202378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5418.637494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5419.262441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5424.221752] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5432.203738] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5436.991523] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5442.883499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5443.433168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5449.531616] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5449.591840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5456.149600] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5460.735750] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5469.461172] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5479.215388] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5488.344161] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5495.781562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5495.782812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5495.792094] LustreError: Skipped 1 previous similar message [ 5495.792714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5495.800049] Lustre: Skipped 1 previous similar message [ 5495.815553] Lustre: Skipped 1 previous similar message [ 5500.899814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5500.915209] Lustre: Skipped 1 previous similar message [ 5501.573587] Lustre: server umount lustre-OST0000 complete [ 5506.018129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.036086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5506.039461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5506.065076] Lustre: Skipped 1 previous similar message [ 5511.141339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5511.157499] Lustre: Skipped 1 previous similar message [ 5516.259495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5516.271995] Lustre: Skipped 1 previous similar message [ 5520.352456] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5520.575475] Lustre: server umount lustre-MDT0000 complete [ 5521.390563] LustreError: 202372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5521.423529] LustreError: 202372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5524.390371] LustreError: 202360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835700 with bad export cookie 2417440698430662887 [ 5524.406312] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5524.419304] LustreError: 202360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5525.230679] Lustre: server umount lustre-MDT0001 complete [ 5537.644199] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5541.056621] Key type lgssc unregistered [ 5541.442710] LNet: 207319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5541.448599] LNetError: 207319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5541.464834] LNet: Removed LNI 192.168.203.159@tcp [ 5542.475481] Key type .llcrypt unregistered [ 5542.479519] Key type ._llcrypt unregistered [ 5561.271709] Key type ._llcrypt registered [ 5561.273172] Key type .llcrypt registered [ 5561.370976] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5562.127405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5562.140736] alg: No test for adler32 (adler32-zlib) [ 5563.169778] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 5563.411364] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5565.072111] Key type lgssc registered [ 5565.969296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5572.399089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5572.410957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.690609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5577.795669] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5583.850626] LustreError: 208710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.865493] LustreError: 208710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5585.335641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5586.035593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5590.331301] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5597.774207] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5603.588581] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5610.143911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5610.533075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5616.659327] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5618.680804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5623.199827] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5639.137559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5639.148688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.151333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5639.160319] Lustre: Skipped 1 previous similar message [ 5642.562828] Lustre: server umount lustre-OST0000 complete [ 5647.329628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5647.339486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.359557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5647.364618] Lustre: Skipped 1 previous similar message [ 5649.378528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5649.378949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5649.401823] Lustre: Skipped 1 previous similar message [ 5652.454930] Lustre: server umount lustre-MDT0000 complete [ 5654.497468] LustreError: 208711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5654.518185] LustreError: 208711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5656.095049] LustreError: 211205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835832 with bad export cookie 4904362821432918846 [ 5656.102221] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5656.105086] LustreError: 211205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5656.444842] Lustre: server umount lustre-MDT0001 complete [ 5666.202453] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5668.978873] Key type lgssc unregistered [ 5669.219406] LNet: 212394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5669.235461] LNetError: 212394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5670.317778] LNet: Removed LNI 192.168.203.159@tcp [ 5671.228820] Key type .llcrypt unregistered [ 5671.230333] Key type ._llcrypt unregistered [ 5689.773021] Key type ._llcrypt registered [ 5689.779213] Key type .llcrypt registered [ 5689.917662] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5690.591997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5690.712550] alg: No test for adler32 (adler32-zlib) [ 5691.780526] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 5692.061363] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5693.776178] Key type lgssc registered [ 5694.733855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5701.677586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5701.704339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.277027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5708.597106] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5713.379335] LustreError: 213659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5713.405728] LustreError: 213659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5717.616789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5717.837122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5722.749774] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5729.115167] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5733.659468] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5741.234780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5741.569590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5747.598739] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5748.797905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5755.069637] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5769.190185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5769.204695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.212475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5769.220345] Lustre: Skipped 1 previous similar message [ 5774.322051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5774.332140] Lustre: Skipped 1 previous similar message [ 5774.927899] Lustre: server umount lustre-OST0000 complete [ 5779.427298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.432305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5779.460029] Lustre: Skipped 1 previous similar message [ 5784.552316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5784.577685] Lustre: Skipped 2 previous similar messages [ 5789.667299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5789.678421] Lustre: Skipped 1 previous similar message [ 5793.248764] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5793.484824] Lustre: server umount lustre-MDT0000 complete [ 5794.792699] LustreError: 214663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.814332] LustreError: 214663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5796.927862] LustreError: 213642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835973 with bad export cookie 17698313152243510360 [ 5796.930818] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5796.936676] LustreError: 213642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5797.375521] Lustre: server umount lustre-MDT0001 complete [ 5808.206674] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5810.937968] Key type lgssc unregistered [ 5811.404935] LNet: 217346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5811.411639] LNetError: 217346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5811.434229] LNet: Removed LNI 192.168.203.159@tcp [ 5812.560953] Key type .llcrypt unregistered [ 5812.563182] Key type ._llcrypt unregistered [ 5832.003736] Key type ._llcrypt registered [ 5832.005791] Key type .llcrypt registered [ 5832.135389] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5833.131244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5833.154421] alg: No test for adler32 (adler32-zlib) [ 5834.204980] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 5834.528715] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5836.213793] Key type lgssc registered [ 5837.136146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5844.814307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5844.842353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5846.250150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5851.042964] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5856.744282] LustreError: 218612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.765296] LustreError: 218612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5859.057513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5859.456246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5864.285738] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5871.907359] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5877.782617] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5884.230712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5884.326340] LustreError: 220432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.591465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5890.517866] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5891.074484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5898.441899] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5908.582709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5908.745839] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5909.094096] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5915.785836] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5922.566363] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5924.735303] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5936.655191] Lustre: Permanently deactivating lustre-OST0001 [ 5936.712434] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5942.361181] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5954.487906] Lustre: Permanently reactivating lustre-OST0001 [ 5954.504927] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5954.509686] Lustre: Skipped 2 previous similar messages [ 5961.290337] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5961.318255] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5961.324615] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5961.334907] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5961.383777] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5966.835561] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5966.838914] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.844492] LustreError: Skipped 1 previous similar message [ 5966.846868] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5966.873115] Lustre: Skipped 2 previous similar messages [ 5966.883573] Lustre: Skipped 1 previous similar message [ 5967.858209] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5967.868578] Lustre: Skipped 1 previous similar message [ 5972.144675] Lustre: server umount lustre-OST0001 complete [ 5972.965266] LustreError: 220433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5972.996577] LustreError: 220433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5977.067049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5977.087840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.107646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5978.081282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5978.089943] LustreError: 220433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5978.147430] LustreError: 220433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5982.771297] Lustre: server umount lustre-OST0000 complete [ 5988.323750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.325418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5988.354241] Lustre: Skipped 2 previous similar messages [ 5988.375582] Lustre: Skipped 3 previous similar messages [ 5993.449539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5993.453697] Lustre: Skipped 1 previous similar message [ 5993.621524] Lustre: server umount lustre-MDT0000 complete [ 5997.499254] LustreError: 218595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836173 with bad export cookie 904281997150776182 [ 5997.503871] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5997.507427] LustreError: 218595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5997.926965] Lustre: server umount lustre-MDT0001 complete [ 6011.902581] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 6015.427867] Key type lgssc unregistered [ 6015.928790] LNet: 223676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6015.942839] LNetError: 223676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6015.978981] LNet: Removed LNI 192.168.203.159@tcp [ 6017.291698] Key type .llcrypt unregistered [ 6017.302665] Key type ._llcrypt unregistered [ 6062.538780] Key type ._llcrypt registered [ 6062.542492] Key type .llcrypt registered [ 6062.643770] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 6064.122388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6064.155239] alg: No test for adler32 (adler32-zlib) [ 6065.370599] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 6065.821273] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 6067.592232] Key type lgssc registered [ 6068.644643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6076.288900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6076.309606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.686566] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6077.722634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6077.883699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6082.145215] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6089.109539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.250798] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6089.290740] Lustre: 225881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6089.704700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6094.839899] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6104.575728] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6111.042549] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6118.717259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.973830] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6119.206833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6124.726977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6125.791637] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6132.840554] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6135.590557] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6141.157475] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6148.802571] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6158.643851] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6170.803916] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6170.814636] Lustre: Skipped 1 previous similar message [ 6193.031282] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6193.047333] Lustre: Skipped 3 previous similar messages [ 6230.430259] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6230.439745] Lustre: Skipped 3 previous similar messages [ 6295.906555] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6295.916380] Lustre: Skipped 7 previous similar messages [ 6303.720468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6303.742378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.765143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6304.249513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6304.261553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.290625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6304.299250] Lustre: Skipped 1 previous similar message [ 6309.555311] Lustre: server umount lustre-OST0000 complete [ 6314.466501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6314.472390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6314.486459] Lustre: Skipped 1 previous similar message [ 6319.589511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6319.606268] Lustre: Skipped 2 previous similar messages [ 6324.708853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6324.717831] Lustre: Skipped 1 previous similar message [ 6328.288157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6328.545492] Lustre: server umount lustre-MDT0000 complete [ 6329.830560] LustreError: 226804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6329.852627] LustreError: 226804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6332.944112] LustreError: 225874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836509 with bad export cookie 14649231974243804445 [ 6332.946983] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6332.955043] LustreError: 225874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6333.383143] Lustre: server umount lustre-MDT0001 complete [ 6347.971945] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 6351.683402] Key type lgssc unregistered [ 6352.226333] LNet: 230679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6352.237860] LNetError: 230679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6352.269873] LNet: Removed LNI 192.168.203.159@tcp [ 6353.569446] Key type .llcrypt unregistered [ 6353.571190] Key type ._llcrypt unregistered [ 6381.004792] Key type ._llcrypt registered [ 6381.013305] Key type .llcrypt registered [ 6381.225525] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 6382.969202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6383.000379] alg: No test for adler32 (adler32-zlib) [ 6384.124272] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 6384.444140] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 6386.216239] Key type lgssc registered [ 6387.490563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6397.106774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6397.158345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6398.807914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6403.033602] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6408.163148] LustreError: 232093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6408.183249] LustreError: 232093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6411.466422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.762402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6416.356902] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6424.406140] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6429.816359] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6437.194762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6437.516939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6443.710158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6444.379966] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6452.195591] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6469.604873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6469.611971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6469.627613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6471.652294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6471.654601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6475.601485] Lustre: server umount lustre-OST0000 complete [ 6481.893506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6481.898723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6481.913958] Lustre: Skipped 1 previous similar message [ 6481.927577] Lustre: Skipped 2 previous similar messages [ 6487.011644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6487.021378] Lustre: Skipped 1 previous similar message [ 6491.106372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6491.109952] Lustre: Skipped 1 previous similar message [ 6494.176415] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6494.492906] Lustre: server umount lustre-MDT0000 complete [ 6496.242598] LustreError: 233098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6496.291890] LustreError: 233098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6498.417292] LustreError: 232075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836674 with bad export cookie 16232611869100254421 [ 6498.429166] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6498.430518] LustreError: 232075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6498.756915] Lustre: server umount lustre-MDT0001 complete [ 6511.441599] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 6514.966662] Key type lgssc unregistered [ 6515.350519] LNet: 235788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6515.361703] LNetError: 235788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6515.384576] LNet: Removed LNI 192.168.203.159@tcp [ 6516.592924] Key type .llcrypt unregistered [ 6516.595397] Key type ._llcrypt unregistered [ 6543.544185] Key type ._llcrypt registered [ 6543.547436] Key type .llcrypt registered [ 6543.680949] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 6545.717494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6545.788995] alg: No test for adler32 (adler32-zlib) [ 6546.916431] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 6547.217049] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 6548.944431] Key type lgssc registered [ 6550.460154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6820.065718] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 6823.065555] Key type lgssc unregistered [ 6823.399539] LNet: 237816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6823.411721] LNetError: 237816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6823.430609] LNet: Removed LNI 192.168.203.159@tcp [ 6824.553467] Key type .llcrypt unregistered [ 6824.554956] Key type ._llcrypt unregistered [ 6852.713916] Key type ._llcrypt registered [ 6852.716402] Key type .llcrypt registered [ 6852.907459] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 6869.989543] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 6871.429879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6871.439257] alg: No test for adler32 (adler32-zlib) [ 6872.556624] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 6872.834500] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 6874.528516] Key type lgssc registered [ 6875.859176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6907.365391] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6761 sec ========== 09:24:42 (1787837082) [ 6909.127463] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:24:44 (1787837084) === [ 6912.661726] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:24:48 (1787837088) === [ 6951.764549] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 6954.431686] Key type lgssc unregistered [ 6954.676249] LNet: 243056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6954.682560] LNetError: 243056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6954.695826] LNet: Removed LNI 192.168.203.159@tcp [ 6955.468674] Key type .llcrypt unregistered [ 6955.472146] Key type ._llcrypt unregistered