[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 522206177 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003390] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009022] pid_max: default: 32768 minimum: 301 [ 0.010202] LSM: Security Framework initializing [ 0.012072] Yama: becoming mindful. [ 0.013060] SELinux: Initializing. [ 0.014119] *** VALIDATE selinux *** [ 0.022725] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027804] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028179] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030046] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032029] *** VALIDATE tmpfs *** [ 0.033405] *** VALIDATE proc *** [ 0.034305] *** VALIDATE cgroup *** [ 0.035017] *** VALIDATE cgroup2 *** [ 0.036289] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040047] Spectre V2 : User space: Vulnerable [ 0.042008] Speculative Store Bypass: Vulnerable [ 0.044938] debug: unmapping init [mem 0xffffffff99459000-0xffffffff99460fff] [ 0.047213] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048883] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049032] ... version: 2 [ 0.050018] ... bit width: 48 [ 0.051026] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054036] ... fixed-purpose events: 3 [ 0.055020] ... event mask: 000000070000000f [ 0.056401] rcu: Hierarchical SRCU implementation. [ 0.058695] smp: Bringing up secondary CPUs ... [ 0.059736] x86: Booting SMP configuration: [ 0.060037] .... node #0, CPUs: #1 #2 #3 [ 0.067118] smp: Brought up 1 node, 4 CPUs [ 0.069019] smpboot: Max logical packages: 1 [ 0.070029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.194034] node 0 deferred pages initialised in 120ms [ 0.199654] devtmpfs: initialized [ 0.201273] x86/mm: Memory block size: 128MB [ 0.205281] gcov: version magic: 0x41383552 [ 0.208097] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.209084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.210337] pinctrl core: initialized pinctrl subsystem [ 0.211267] [ 0.211742] ************************************************************* [ 0.212016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.213019] ** ** [ 0.214020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.215018] ** ** [ 0.216020] ** This means that this kernel is built to expose internal ** [ 0.217017] ** IOMMU data structures, which may compromise security on ** [ 0.218015] ** your system. ** [ 0.219018] ** ** [ 0.220024] ** If you see this message and you are not debugging the ** [ 0.221016] ** kernel, report this immediately to your vendor! ** [ 0.222019] ** ** [ 0.223020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.224012] ************************************************************* [ 0.225765] NET: Registered protocol family 16 [ 0.226495] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.227082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.228082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.229581] cpuidle: using governor menu [ 0.231956] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.233602] PCI: Using configuration type 1 for base access [ 0.236143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.242159] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.244023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.247163] cryptd: max_cpu_qlen set to 1000 [ 0.251816] ACPI: Added _OSI(Module Device) [ 0.253019] ACPI: Added _OSI(Processor Device) [ 0.255017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.257015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.258000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.262492] ACPI: Interpreter enabled [ 0.263000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.263000] ACPI: Using IOAPIC for interrupt routing [ 0.265135] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.268489] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.278793] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.282057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.285024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.289123] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.294430] acpiphp: Slot [2] registered [ 0.296169] acpiphp: Slot [3] registered [ 0.298119] acpiphp: Slot [4] registered [ 0.300147] acpiphp: Slot [5] registered [ 0.301172] acpiphp: Slot [6] registered [ 0.303186] acpiphp: Slot [7] registered [ 0.304157] acpiphp: Slot [8] registered [ 0.306145] acpiphp: Slot [9] registered [ 0.308180] acpiphp: Slot [10] registered [ 0.310143] acpiphp: Slot [11] registered [ 0.312112] acpiphp: Slot [12] registered [ 0.313213] acpiphp: Slot [13] registered [ 0.315121] acpiphp: Slot [14] registered [ 0.317145] acpiphp: Slot [15] registered [ 0.319121] acpiphp: Slot [16] registered [ 0.320112] acpiphp: Slot [17] registered [ 0.322135] acpiphp: Slot [18] registered [ 0.323180] acpiphp: Slot [19] registered [ 0.325152] acpiphp: Slot [20] registered [ 0.327132] acpiphp: Slot [21] registered [ 0.329135] acpiphp: Slot [22] registered [ 0.330139] acpiphp: Slot [23] registered [ 0.332121] acpiphp: Slot [24] registered [ 0.334119] acpiphp: Slot [25] registered [ 0.335206] acpiphp: Slot [26] registered [ 0.337140] acpiphp: Slot [27] registered [ 0.339121] acpiphp: Slot [28] registered [ 0.340166] acpiphp: Slot [29] registered [ 0.342130] acpiphp: Slot [30] registered [ 0.344145] acpiphp: Slot [31] registered [ 0.345073] PCI host bridge to bus 0000:00 [ 0.347029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.350033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.351000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.351029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.354036] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.357045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.359297] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.363418] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.369387] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.380021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.386067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.389041] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.391022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.396028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.399535] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.401746] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.404061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.407665] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.411018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.422017] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.427858] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.431000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.439019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.451023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.467022] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.483525] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.490018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.497018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.511035] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.517023] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.528018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.539021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.551164] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.563821] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.571021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.579021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.589020] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.599912] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.613019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.631021] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.642032] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.659022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.672028] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.686021] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.697172] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.699372] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.702425] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.705416] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.707282] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.711137] iommu: Default domain type: Passthrough [ 0.716506] SCSI subsystem initialized [ 0.718288] ACPI: bus type USB registered [ 0.719706] usbcore: registered new interface driver usbfs [ 0.720000] usbcore: registered new interface driver hub [ 0.720108] usbcore: registered new device driver usb [ 0.721117] pps_core: LinuxPPS API ver. 1 registered [ 0.724015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.726115] PTP clock support registered [ 0.729085] EDAC MC: Ver: 3.0.0 [ 0.732267] PCI: Using ACPI for IRQ routing [ 0.734876] NetLabel: Initializing [ 0.736012] NetLabel: domain hash size = 128 [ 0.741019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.743158] NetLabel: unlabeled traffic allowed by default [ 0.746099] vgaarb: loaded [ 0.748331] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.751018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.760000] clocksource: Switched to clocksource kvm-clock [ 0.880107] VFS: Disk quotas dquot_6.6.0 [ 0.881529] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.883894] *** VALIDATE ramfs *** [ 0.885131] *** VALIDATE hugetlbfs *** [ 0.886738] pnp: PnP ACPI init [ 0.888812] pnp: PnP ACPI: found 6 devices [ 0.905756] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.908190] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.910420] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.912788] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.915516] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.918251] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.921042] NET: Registered protocol family 2 [ 0.923531] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.928917] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.932785] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.940228] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.943318] TCP: Hash tables configured (established 65536 bind 65536) [ 0.946392] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.949512] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951855] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.954689] NET: Registered protocol family 1 [ 0.957224] RPC: Registered named UNIX socket transport module. [ 0.959945] RPC: Registered udp transport module. [ 0.961363] RPC: Registered tcp transport module. [ 0.962728] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.964555] NET: Registered protocol family 44 [ 0.965765] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.967291] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.968830] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.970614] PCI: CLS 0 bytes, default 64 [ 0.971921] Unpacking initramfs... [ 2.684847] debug: unmapping init [mem 0xffff9dba3cc54000-0xffff9dba3ffbffff] [ 2.691690] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.695391] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.699756] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.223982] Initialise system trusted keyrings [ 3.226023] Key type blacklist registered [ 3.227725] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.236873] zbud: loaded [ 3.239646] *** VALIDATE nfs *** [ 3.240863] *** VALIDATE nfs4 *** [ 3.242565] pstore: using deflate compression [ 3.246131] Platform Keyring initialized [ 3.390475] NET: Registered protocol family 38 [ 3.392814] Key type asymmetric registered [ 3.394800] Asymmetric key parser 'x509' registered [ 3.397192] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.401275] io scheduler mq-deadline registered [ 3.404477] io scheduler kyber registered [ 3.417761] io scheduler bfq registered [ 3.421346] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.425605] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.429714] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.433818] ACPI: Power Button [PWRF] [ 3.536643] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.641420] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.871875] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.980097] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.202099] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.235277] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.266198] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.274421] Non-volatile memory driver v1.3 [ 4.276336] Linux agpgart interface v0.103 [ 4.314230] virtio_blk virtio1: [vda] 67984 512-byte logical blocks (34.8 MB/33.2 MiB) [ 4.319158] vda: detected capacity change from 0 to 34807808 [ 4.343946] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.348212] vdb: detected capacity change from 0 to 1073741824 [ 4.365257] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.368539] vdc: detected capacity change from 0 to 2621440000 [ 4.393651] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.399939] vdd: detected capacity change from 0 to 2621440000 [ 4.416352] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.419895] vde: detected capacity change from 0 to 4294967296 [ 4.444704] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.447544] vdf: detected capacity change from 0 to 4294967296 [ 4.457307] libphy: Fixed MDIO Bus: probed [ 4.461921] usbcore: registered new interface driver usbserial_generic [ 4.468143] usbserial: USB Serial support registered for generic [ 4.470447] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.479488] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.484805] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.487206] mousedev: PS/2 mouse device common for all mice [ 4.491856] rtc_cmos 00:05: RTC can wake from S4 [ 4.496478] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.500610] rtc_cmos 00:05: registered as rtc0 [ 4.514730] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.521232] intel_pstate: CPU model not supported [ 4.523777] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.535173] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.538390] hid: raw HID events driver (C) Jiri Kosina [ 4.540464] usbcore: registered new interface driver usbhid [ 4.542199] usbhid: USB HID core driver [ 4.543603] drop_monitor: Initializing network drop monitor service [ 4.545800] Initializing XFRM netlink socket [ 4.547624] NET: Registered protocol family 10 [ 4.552822] Segment Routing with IPv6 [ 4.554054] NET: Registered protocol family 17 [ 4.562554] mpls_gso: MPLS GSO support [ 4.570876] RAS: Correctable Errors collector initialized. [ 4.573427] AVX version of gcm_enc/dec engaged. [ 4.575028] AES CTR mode by8 optimization enabled [ 4.715825] sched_clock: Marking stable (4715801783, 0)->(5873018363, -1157216580) [ 4.730530] registered taskstats version 1 [ 4.732573] Loading compiled-in X.509 certificates [ 4.734396] zswap: loaded using pool lzo/zbud [ 4.774532] Key type big_key registered [ 4.800605] Key type encrypted registered [ 4.802849] ima: No TPM chip found, activating TPM-bypass! [ 4.805115] ima: Allocated hash algorithm: sha1 [ 4.807200] ima: No architecture policies found [ 4.809228] evm: Initialising EVM extended attributes: [ 4.811323] evm: security.selinux [ 4.812564] evm: security.ima [ 4.813791] evm: security.capability [ 4.815363] evm: HMAC attrs: 0x1 [ 4.819215] rtc_cmos 00:05: setting system clock to 2025-10-24 13:10:06 UTC (1761311406) [ 4.826459] debug: unmapping init [mem 0xffffffff9a403000-0xffffffff9a5fffff] [ 4.829883] debug: unmapping init [mem 0xffffffff99182000-0xffffffff99458fff] [ 4.838204] Write protecting the kernel read-only data: 28672k [ 4.842329] debug: unmapping init [mem 0xffffffff97803000-0xffffffff979fffff] [ 4.846415] debug: unmapping init [mem 0xffffffff98114000-0xffffffff981fffff] [ 4.909872] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.921806] systemd[1]: Detected virtualization kvm. [ 4.924942] systemd[1]: Detected architecture x86-64. [ 4.926969] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.963388] systemd[1]: No hostname configured. [ 4.965700] systemd[1]: Set hostname to . [ 4.968684] random: systemd: uninitialized urandom read (16 bytes read) [ 4.976321] systemd[1]: Initializing machine ID from random generator. [ 5.064692] random: ln: uninitialized urandom read (6 bytes read) [ 5.238723] random: systemd: uninitialized urandom read (16 bytes read) [ 5.247696] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.254262] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.263401] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.363739] device-mapper: uevent: version 1.0.3 [ 6.366685] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. [ 6.723271] random: fast init done Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.446295] virtio_net virtio0 ens2: renamed from eth0 [ 7.763022] scsi host0: ata_piix [ 7.772652] scsi host1: ata_piix [ 7.773797] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.785399] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.327080] random: crng init done [ 12.328391] random: 7 urandom warning(s) missed due to ratelimiting [ 12.583393] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.152953] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.919403] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.272732] SELinux: Disabled at runtime. [ 16.347648] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.359111] systemd[1]: Detected virtualization kvm. [ 16.361312] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.128777] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.132678] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.144313] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.150353] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.154678] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.173417] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.184338] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX M[ 17.364658] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS essage Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 17.952650] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 18.612766] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.750333] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.383866] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.421715] EDAC sbridge: Ver: 1.1.2 [ 22.414052] Key type dns_resolver registered [ 22.805865] NFS: Registering the id_resolver key type [ 22.807817] Key type id_resolver registered [ 22.810509] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg259-server login: [ 58.923911] libcfs: loading out-of-tree module taints kernel. [ 58.980958] alg: No test for adler32 (adler32-zlib) [ 59.738725] Key type ._llcrypt registered [ 59.740234] Key type .llcrypt registered [ 59.848662] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 77.680932] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 79.475357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.284682] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 80.878556] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 80.882721] LNet: Accept secure, port 988 [ 82.623536] Key type lgssc registered [ 83.805576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.365916] hrtimer: interrupt took 10013138 ns [ 100.100451] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 101.921440] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.445518] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.338081] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.022142] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.376539] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 141.692912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.797833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.984694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 143.005331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 143.071113] Lustre: lustre-MDT0000: new disk, initializing [ 143.132761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.142238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.965380] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 156.175885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.244673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.319720] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 156.343121] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 156.345377] Lustre: Skipped 1 previous similar message [ 156.403145] Lustre: lustre-MDT0001: new disk, initializing [ 156.465481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 156.492181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 156.503242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 159.796587] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 172.159561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.230990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 172.235697] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 172.434499] Lustre: lustre-OST0000: new disk, initializing [ 172.446984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 172.497391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 175.975725] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 179.740719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.755303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 188.725182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.810356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 188.823289] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 189.012774] Lustre: lustre-OST0001: new disk, initializing [ 189.025902] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 189.143919] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.823831] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 198.697669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 198.710900] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 202.006629] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 206.504319] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 214.985254] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing check_logdir /tmp/testlogs/ [ 219.203788] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing yml_node [ 223.214651] Lustre: DEBUG MARKER: Client: 2.15.7.10 [ 225.678826] Lustre: DEBUG MARKER: MDS: 2.15.7.10 [ 227.826419] Lustre: DEBUG MARKER: OSS: 2.15.7.10 [ 229.377837] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 09:13:49 EDT 2025 [ 236.866810] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 238.184770] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 253.924420] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 253.939454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.996248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 254.944883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 254.949222] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 258.702887] Lustre: server umount lustre-MDT0000 complete [ 260.065958] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 262.503293] LustreError: 6245:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311664 with bad export cookie 9516372721556662260 [ 262.505642] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 262.509969] LustreError: 6245:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 265.183838] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 265.188307] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.190766] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 265.190773] Lustre: Skipped 3 previous similar messages [ 265.204892] LustreError: Skipped 4 previous similar messages [ 265.250668] Lustre: Skipped 3 previous similar messages [ 267.232313] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 267.234359] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 267.241961] Lustre: Skipped 2 previous similar messages [ 267.254165] LustreError: Skipped 1 previous similar message [ 269.235383] Lustre: server umount lustre-MDT0001 complete [ 272.842183] Lustre: server umount lustre-OST0000 complete [ 276.553779] Lustre: server umount lustre-OST0001 complete [ 282.475491] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 289.616291] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 297.399787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 304.034729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 310.526053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 316.405072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 322.428920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 322.495667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 322.753834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 322.788776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 322.898932] Lustre: lustre-MDT0000: new disk, initializing [ 323.037152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 323.075849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 326.299620] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 334.790191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 334.835469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 334.917322] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 334.960935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 334.964232] Lustre: Skipped 1 previous similar message [ 335.020322] Lustre: lustre-MDT0001: new disk, initializing [ 335.118910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 335.161610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 335.172694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 338.995475] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 347.422551] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 351.904119] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 357.044798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 357.112831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 357.117668] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 357.277297] Lustre: lustre-OST0000: new disk, initializing [ 357.280678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 357.346724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 359.071212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 359.085680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 360.406473] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 367.038148] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 370.335724] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 370.548098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 374.376873] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 374.570622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 379.873032] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 379.877203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.878713] LustreError: Skipped 1 previous similar message [ 379.884355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 379.894082] Lustre: Skipped 1 previous similar message [ 382.384835] Lustre: server umount lustre-OST0000 complete [ 386.529463] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 386.535959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.543655] Lustre: Skipped 1 previous similar message [ 390.113872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 390.118391] Lustre: Skipped 2 previous similar messages [ 391.446655] Lustre: server umount lustre-MDT0000 complete [ 394.292511] LustreError: 16427:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311795 with bad export cookie 9516372721556663716 [ 394.294129] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 394.300858] LustreError: 16427:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 398.358807] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 09:16:38 (1761311798) [ 402.754038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.044617] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 403.104163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.575457] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 408.545021] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.302257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.585754] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 422.274149] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 427.290141] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 433.877319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 434.098653] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.821621] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 446.596743] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 450.329571] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 457.187670] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.189542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 457.201705] Lustre: Skipped 2 previous similar messages [ 457.212912] Lustre: Skipped 2 previous similar messages [ 460.826140] Lustre: server umount lustre-MDT0000 complete [ 460.829566] Lustre: Skipped 1 previous similar message [ 462.304246] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.313276] LustreError: Skipped 2 previous similar messages [ 464.405966] LustreError: 18726:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311866 with bad export cookie 9516372721556664570 [ 464.409278] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 464.419736] LustreError: 18726:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 473.567374] Lustre: 21853:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311869/real 1761311869] req@00000000a3553422 x1846868948627648/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311875 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 491.898358] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 499.745890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.044426] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.098920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.102209] Lustre: Skipped 2 previous similar messages [ 503.604348] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 509.974875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.643659] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 520.909640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 524.784429] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 532.115156] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 532.188869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 532.194957] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 532.296384] Lustre: lustre-OST0001: new disk, initializing [ 532.298885] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 534.049961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 534.056951] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 536.328486] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 545.564574] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 548.190914] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 564.715140] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.720364] Lustre: Skipped 1 previous similar message [ 564.723212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 567.916466] Lustre: server umount lustre-MDT0000 complete [ 567.918594] Lustre: Skipped 2 previous similar messages [ 569.825074] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.852952] LustreError: Skipped 4 previous similar messages [ 571.810721] LustreError: 23217:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311973 with bad export cookie 9516372721556665417 [ 571.818137] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 571.821579] LustreError: 23217:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 582.047165] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311976/real 1761311976] req@00000000c7facc8c x1846868948657152/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311983 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 584.096231] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311978/real 1761311978] req@00000000bd0872bc x1846868948657344/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311985 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 584.107672] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 599.647528] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 607.031144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.802940] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 617.173744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.025593] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 625.628795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 629.176211] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 636.196256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 636.412688] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 636.420417] Lustre: Skipped 6 previous similar messages [ 639.778549] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 645.487138] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 647.913758] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 647.916935] Lustre: Skipped 1 previous similar message [ 657.893332] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.900142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 657.911092] Lustre: Skipped 7 previous similar messages [ 657.918644] Lustre: Skipped 5 previous similar messages [ 663.126246] Lustre: server umount lustre-MDT0000 complete [ 663.128487] Lustre: Skipped 3 previous similar messages [ 666.101062] LustreError: 28461:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312067 with bad export cookie 9516372721556666663 [ 666.105092] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 666.134936] LustreError: 28461:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 675.167164] Lustre: 3362:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312069/real 1761312069] req@000000005b436d74 x1846868948679360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312076 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 675.179689] Lustre: 3362:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 686.339949] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 09:21:26 (1761312086) [ 705.254634] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 711.765243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.101675] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.107730] LustreError: Skipped 3 previous similar messages [ 715.382061] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 721.321951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.987606] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 730.382612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 733.705434] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 739.958577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 743.090933] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 749.159889] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 752.074501] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 762.850961] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 762.857981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.876977] Lustre: Skipped 1 previous similar message [ 769.207095] LustreError: 34298:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312170 with bad export cookie 9516372721556667727 [ 769.215518] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.234187] LustreError: 34298:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 778.719215] Lustre: 37868:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312174/real 1761312174] req@00000000acbf5139 x1846868948701504/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312180 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 778.729413] Lustre: 37868:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 787.687805] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 09:23:07 (1761312187) [ 798.826894] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 805.540153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.244395] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 815.527743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.507172] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 825.796329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 828.876527] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 834.653218] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 837.891696] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 843.974295] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 856.551564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 856.558325] Lustre: Skipped 12 previous similar messages [ 861.658854] Lustre: server umount lustre-MDT0000 complete [ 861.664446] Lustre: Skipped 7 previous similar messages [ 864.639944] LustreError: 41076:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312266 with bad export cookie 9516372721556668805 [ 864.645619] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 877.184544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 882.146280] Lustre: Evicted from MGS (at 192.168.202.159@tcp) after server handle changed from 0x8410f292029a4185 to 0x8410f292029a4608 [ 882.156053] Lustre: MGC192.168.202.159@tcp: Connection restored to (at 0@lo) [ 882.383197] BUG: spinlock bad magic on CPU#2, lctl/43404 [ 882.391340] lock: 0xffff9dba9018a500, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 882.394754] CPU: 2 PID: 43404 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 882.398285] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 882.401831] Call Trace: [ 882.404537] ? dump_stack+0xbb/0x10e [ 882.406583] ? spin_dump.cold.3+0x32/0x65 [ 882.408857] ? do_raw_spin_lock+0xce/0x190 [ 882.412737] ? _raw_spin_lock+0x21/0x30 [ 882.414010] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 882.417326] ? __seccomp_filter+0x4b/0x680 [ 882.419074] ? full_proxy_write+0x5e/0xa0 [ 882.421335] ? __vfs_write+0x1c/0x60 [ 882.422867] ? vfs_write+0xd8/0x2b0 [ 882.424834] ? ksys_write+0x66/0x120 [ 882.426312] ? __x64_sys_write+0x1e/0x30 [ 882.430461] ? do_syscall_64+0xc1/0x440 [ 882.431961] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 885.074760] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 892.059877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.895069] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 896.421774] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 900.519420] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 900.521893] Lustre: Skipped 1 previous similar message [ 901.731145] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 903.043901] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 907.361810] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 908.823213] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 912.866131] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 912.875434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.891580] Lustre: Skipped 10 previous similar messages [ 922.609288] LustreError: 39474:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312324 with bad export cookie 9516372721556669960 [ 922.610273] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 922.615912] LustreError: 39474:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 932.831152] Lustre: 45625:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312328/real 1761312328] req@0000000014679fa4 x1846868948741248/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312334 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 932.848297] Lustre: 45625:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 941.224494] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 09:25:41 (1761312341) [ 951.153776] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 957.728073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.050822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.053858] Lustre: Skipped 10 previous similar messages [ 960.573374] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 965.895473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.154884] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 975.318750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 979.316605] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 981.731530] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 981.750040] LustreError: Skipped 31 previous similar messages [ 986.987498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 990.298636] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 996.149795] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 999.064709] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 999.071621] Lustre: Skipped 2 previous similar messages [ 1018.014221] LustreError: 47233:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312419 with bad export cookie 9516372721556670744 [ 1018.019453] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1025.951164] Lustre: 3362:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312420/real 1761312420] req@00000000a1a0f771 x1846868948766144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312427 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1025.963313] Lustre: 3362:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1036.067504] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 09:27:16 (1761312436) [ 1037.918778] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1039.111746] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 09:27:19 (1761312439) [ 1047.284125] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 1052.843903] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1059.330661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1063.881492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1068.653814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1068.710473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.865325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1068.920930] Lustre: lustre-MDT0000: new disk, initializing [ 1069.020188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1072.787340] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1081.292105] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1085.994802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1086.054381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1086.058916] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1089.896575] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1097.596622] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1099.415411] Lustre: DEBUG MARKER: create test files [ 1118.690941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1118.692844] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1118.698553] Lustre: Skipped 18 previous similar messages [ 1119.102836] Lustre: server umount lustre-OST0000 complete [ 1119.105918] Lustre: Skipped 9 previous similar messages [ 1127.391156] Lustre: 55533:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312523/real 1761312523] req@00000000f86ef43f x1846868948774656/t0(0) o251->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761312529 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1127.409910] Lustre: 55533:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1135.924934] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1137.961399] Key type lgssc unregistered [ 1138.222903] LNet: 56137:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.231258] LNet: Removed LNI 192.168.202.159@tcp [ 1138.905327] Key type .llcrypt unregistered [ 1138.907354] Key type ._llcrypt unregistered [ 1160.416774] alg: No test for adler32 (adler32-zlib) [ 1161.178701] Key type ._llcrypt registered [ 1161.179903] Key type .llcrypt registered [ 1161.278683] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1162.067772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1162.524582] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 1162.696049] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1162.698805] LNet: Accept secure, port 988 [ 1164.367160] Key type lgssc registered [ 1165.274806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1171.973570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1173.384969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1177.296879] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1184.891218] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1190.547189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1190.824670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1194.466650] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1201.514976] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1204.083598] Lustre: lustre-OST0000: deleting orphan objects from 0x0:102 to 0x0:129 [ 1204.164692] Lustre: 58977:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1204.178581] Lustre: 58977:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1204.182238] Lustre: 58977:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1204.186839] Lustre: 58977:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1204.191137] Lustre: 58977:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1204.195377] Lustre: 58977:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1211.664736] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1216.259124] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1224.672241] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1224.678977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.690630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1226.652816] Lustre: server umount lustre-OST0000 complete [ 1235.424389] Lustre: 59991:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312631/real 1761312631] req@000000005a4388a7 x1846870083115328/t0(0) o251->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761312637 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1235.605967] Lustre: server umount lustre-MDT0000 complete [ 1246.114986] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1248.545308] Key type lgssc unregistered [ 1248.753300] LNet: 60596:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.760428] LNet: Removed LNI 192.168.202.159@tcp [ 1249.500346] Key type .llcrypt unregistered [ 1249.503063] Key type ._llcrypt unregistered [ 1271.702987] alg: No test for adler32 (adler32-zlib) [ 1272.457414] Key type ._llcrypt registered [ 1272.460471] Key type .llcrypt registered [ 1272.557470] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1273.225547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1273.438967] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 1273.574959] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1273.578905] LNet: Accept secure, port 988 [ 1275.207151] Key type lgssc registered [ 1275.945318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1281.966728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.898398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.398228] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1296.844877] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1302.802106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1303.159446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.771953] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1312.560179] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1313.389610] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 1313.461585] Lustre: 63533:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1313.470994] Lustre: 63533:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1313.474992] Lustre: 63533:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1313.481534] Lustre: 63533:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1313.487209] Lustre: 63533:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1313.495952] Lustre: 63533:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1322.148988] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1329.122364] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1329.133469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.147510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1334.240810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1334.519686] Lustre: server umount lustre-OST0000 complete [ 1344.991165] Lustre: 64498:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312740/real 1761312740] req@00000000338e03d2 x1846870199507072/t0(0) o251->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761312746 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1345.197676] Lustre: server umount lustre-MDT0000 complete [ 1355.431247] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1357.405155] Key type lgssc unregistered [ 1357.592288] LNet: 65101:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.599980] LNet: Removed LNI 192.168.202.159@tcp [ 1358.111711] Key type .llcrypt unregistered [ 1358.113672] Key type ._llcrypt unregistered [ 1371.636980] alg: No test for adler32 (adler32-zlib) [ 1372.389447] Key type ._llcrypt registered [ 1372.392688] Key type .llcrypt registered [ 1372.476502] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 1381.733232] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1382.298884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1382.555451] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 1382.711079] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1382.715062] LNet: Accept secure, port 988 [ 1384.327488] Key type lgssc registered [ 1384.979712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1392.165408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1398.046110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1402.879927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1408.285211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1412.732419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1412.772151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1413.916792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1413.933720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1413.983421] Lustre: lustre-MDT0000: new disk, initializing [ 1414.050719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.066121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1416.708426] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1424.597728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1424.644966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.709171] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1424.741140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1424.743849] Lustre: Skipped 1 previous similar message [ 1424.791670] Lustre: lustre-MDT0001: new disk, initializing [ 1424.859780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1424.891508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1424.903183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1428.024647] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1436.157112] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1441.032253] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1446.324604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1446.360536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1446.364910] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1446.557803] Lustre: lustre-OST0000: new disk, initializing [ 1446.561622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1446.633043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1449.936326] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1455.642302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1455.656144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1457.542634] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1461.028327] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1461.207755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1465.503174] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1465.683330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1470.946071] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1470.947193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.955744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1473.569395] Lustre: server umount lustre-OST0000 complete [ 1481.186039] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1481.186955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.198402] Lustre: Skipped 1 previous similar message [ 1481.203257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.207900] Lustre: Skipped 2 previous similar messages [ 1482.836275] Lustre: server umount lustre-MDT0000 complete [ 1486.267860] LustreError: 68373:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312887 with bad export cookie 13590857697725232532 [ 1486.272043] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1486.278066] LustreError: 68373:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1486.595079] Lustre: server umount lustre-MDT0001 complete [ 1492.991919] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 09:34:53 (1761312893) [ 1499.668649] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1508.173528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1523.143917] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 1528.348607] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1533.869504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1538.091645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1542.002610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1546.305397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1550.929586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1550.976122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1551.107665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1551.142433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1551.215627] Lustre: lustre-MDT0000: new disk, initializing [ 1551.270363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.289685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1554.279072] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1561.564366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1561.608220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.657965] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1561.690136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1561.699672] Lustre: Skipped 1 previous similar message [ 1561.753911] Lustre: lustre-MDT0001: new disk, initializing [ 1561.819905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1561.840158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1561.844984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1564.864266] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1572.424712] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1577.282199] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1582.681416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1582.743930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1582.749323] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1582.921225] Lustre: lustre-OST0000: new disk, initializing [ 1582.923925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1583.014444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.339678] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1589.816185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1589.821604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1593.882679] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1597.407172] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1597.599956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1601.556804] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1601.752791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1605.101208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.106494] Lustre: Skipped 1 previous similar message [ 1605.109508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1609.754228] Lustre: server umount lustre-OST0000 complete [ 1613.282094] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1613.296365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.308012] Lustre: Skipped 1 previous similar message [ 1613.314961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1613.328611] Lustre: Skipped 1 previous similar message [ 1618.680298] Lustre: server umount lustre-MDT0000 complete [ 1620.448235] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.460179] LustreError: Skipped 1 previous similar message [ 1621.837799] LustreError: 77515:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313023 with bad export cookie 13590857697725233386 [ 1621.845469] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1621.853189] LustreError: 77515:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1627.493632] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 09:37:07 (1761313027) [ 1631.618028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.856439] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1631.894525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.191147] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1637.348605] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.134935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.119633] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1651.887188] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1657.400230] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1662.450847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1665.723727] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1670.694764] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1673.320466] Lustre: 83955:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting 3c3560fe-acab-4073-966e-e9d07365dc9f at adminstrative request [ 1676.256816] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.268664] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1681.376361] Lustre: MGS: Client 3c3560fe-acab-4073-966e-e9d07365dc9f (at 0@lo) reconnecting [ 1681.382329] Lustre: Evicted from MGS (at 192.168.202.159@tcp) after server handle changed from 0xbc9c6d1191cbbc40 to 0xbc9c6d1191cbbed2 [ 1681.395886] Lustre: MGC192.168.202.159@tcp: Connection restored to 192.168.202.159@tcp (at 0@lo) [ 1708.094804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1708.145927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1708.150399] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1708.265439] Lustre: lustre-OST0001: new disk, initializing [ 1708.271796] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1708.336259] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1708.346470] Lustre: Skipped 2 previous similar messages [ 1711.436626] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1718.415211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1718.426925] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1719.776766] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1723.880843] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1723.886248] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.895696] Lustre: Skipped 1 previous similar message [ 1723.900766] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1723.904777] Lustre: Skipped 3 previous similar messages [ 1724.258804] Lustre: server umount lustre-OST0001 complete [ 1724.267653] Lustre: Skipped 1 previous similar message [ 1728.993643] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1729.015327] LustreError: Skipped 1 previous similar message [ 1738.215134] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1738.219068] LustreError: Skipped 1 previous similar message [ 1738.221359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.226717] Lustre: Skipped 3 previous similar messages [ 1738.231472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1738.234263] Lustre: Skipped 3 previous similar messages [ 1743.395021] Lustre: server umount lustre-MDT0000 complete [ 1743.396690] Lustre: Skipped 1 previous similar message [ 1744.352427] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1744.382091] LustreError: Skipped 1 previous similar message [ 1747.638670] LustreError: 84352:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313149 with bad export cookie 13590857697725234898 [ 1747.646383] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1747.652146] LustreError: 84352:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1755.469087] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 09:39:15 (1761313155) [ 1757.019247] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1758.426608] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 09:39:18 (1761313158) [ 1759.463338] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1760.610222] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 09:39:20 (1761313160) [ 1761.829743] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1763.065787] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 09:39:23 (1761313163) [ 1764.033523] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1767.515570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1768.447565] loop: module loaded [ 1768.469550] loop0: detected capacity change from 0 to 4294967296 [ 1768.565972] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1769.838818] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1772.560381] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1776.285041] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1783.638973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1787.633213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1791.685697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1795.638222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1801.627927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1801.691959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1801.696963] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1801.718254] LustreError: 88404:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1801.730958] LustreError: 88404:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1801.738447] LustreError: 88404:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1801.745523] LustreError: 88404:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1816.456490] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 1823.262436] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1830.452119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1836.054101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1841.455864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1847.652674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1853.860699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1853.934271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.161951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1854.190178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1854.264779] Lustre: lustre-MDT0000: new disk, initializing [ 1854.398472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.424822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1859.217831] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1870.448449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1870.513684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.632568] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1870.879822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1875.049976] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1883.948554] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1889.033467] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1895.113210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1895.212579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1895.221396] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1899.324235] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1904.217059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1907.166984] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1910.651799] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1910.949192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1914.451614] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1914.586584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1919.968472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.968786] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1919.972090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1919.972100] Lustre: Skipped 2 previous similar messages [ 1919.973843] Lustre: Skipped 2 previous similar messages [ 1919.989176] LustreError: Skipped 1 previous similar message [ 1930.213435] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1930.383808] Lustre: server umount lustre-OST0000 complete [ 1930.389880] Lustre: Skipped 1 previous similar message [ 1948.127470] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1950.694097] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.703747] LustreError: Skipped 1 previous similar message [ 1951.302227] LustreError: 91258:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313352 with bad export cookie 13590857697725235479 [ 1951.320750] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1951.379456] LustreError: 91258:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1960.296715] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 09:42:39 (1761313359) [ 1965.860295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.992844] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1971.681706] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.697028] LustreError: Skipped 1 previous similar message [ 1978.359407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.555702] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 1989.266618] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1994.682513] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2000.814246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2001.223449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2001.226929] Lustre: Skipped 4 previous similar messages [ 2004.893612] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2011.762615] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2018.083191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2018.170803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2018.177757] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2018.318583] Lustre: lustre-OST0001: new disk, initializing [ 2018.320249] Lustre: Skipped 2 previous similar messages [ 2018.328744] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2018.339067] Lustre: Skipped 3 previous similar messages [ 2022.841238] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2023.993329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2024.000479] Lustre: Skipped 2 previous similar messages [ 2024.005280] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2030.923219] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2047.966922] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2067.405539] Lustre: Failing over lustre-MDT0000 [ 2067.759379] Lustre: server umount lustre-MDT0000 complete [ 2067.765056] Lustre: Skipped 2 previous similar messages [ 2068.787113] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.793578] LustreError: Skipped 2 previous similar messages [ 2069.985487] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.999932] Lustre: Skipped 4 previous similar messages [ 2077.151622] Lustre: 66600:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313471/real 1761313471] req@00000000be4140e9 x1846870313912512/t0(0) o400->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761313478 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2077.165761] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2090.416040] LDISKFS-fs (dm-0): recovery complete [ 2090.423144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.542049] Lustre: Evicted from MGS (at 192.168.202.159@tcp) after server handle changed from 0xbc9c6d1191cbc46d to 0xbc9c6d1191cc5250 [ 2093.556807] Lustre: MGC192.168.202.159@tcp: Connection restored to (at 0@lo) [ 2093.833668] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2097.436294] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2097.439642] Lustre: 97184:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2097.790330] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2099.170485] Lustre: 95820:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2099.176495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2099.182322] Lustre: 95820:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 1 previous similar message [ 2099.291765] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2099.616060] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2099.937902] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2099.945045] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2100.265067] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2100.274301] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2100.287220] Lustre: 100997:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2100.307485] Lustre: 100997:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 7 previous similar messages [ 2100.945528] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2100.960893] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 2101.280664] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2101.297972] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 2102.585411] Lustre: 100997:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2102.595768] Lustre: 100997:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 6 previous similar messages [ 2103.237887] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2103.246519] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2103.581435] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2103.587592] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2106.752038] Lustre: 100997:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2106.772959] Lustre: 100997:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages [ 2107.418986] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2107.438578] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2107.759258] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2107.767387] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2115.057655] Lustre: 100997:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2115.066326] Lustre: 100997:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 25 previous similar messages [ 2115.708593] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2115.712436] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 2116.034679] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2116.037829] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 2131.322582] Lustre: 100997:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2131.334216] Lustre: 100997:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 50 previous similar messages [ 2132.019090] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2132.030924] LustreError: 100997:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2132.351138] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2132.363340] LustreError: 100997:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2157.023196] Lustre: 100997:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2157.041656] Lustre: 100997:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2157.050504] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2157.057278] LustreError: 100997:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 2157.066737] Lustre: 100997:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@00000000ff4b44c0 x1846870304387648/t0(8589934778) o35->cb6f23bb-e8dd-42e7-8d11-5e003450cc5f@192.168.202.59@tcp:489/0 lens 392/0 e 0 to 0 dl 1761313564 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 2157.100405] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 2157.103144] Lustre: Skipped 2 previous similar messages [ 2157.109547] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2157.152380] Lustre: lustre-OST0001: deleting orphan objects from 0x0:48 to 0x0:545 [ 2157.155985] Lustre: lustre-OST0000: deleting orphan objects from 0x0:79 to 0x0:577 [ 2165.740072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.743895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2165.751436] Lustre: Skipped 2 previous similar messages [ 2165.763022] Lustre: Skipped 12 previous similar messages [ 2175.973673] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.980064] LustreError: Skipped 43 previous similar messages [ 2188.255157] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2192.867082] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2198.554334] Lustre: server umount lustre-MDT0000 complete [ 2198.612755] Lustre: Skipped 2 previous similar messages [ 2202.104699] LustreError: 95801:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313603 with bad export cookie 13590857697725272656 [ 2202.107106] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2202.118024] LustreError: 95801:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2210.242776] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 09:46:49 (1761313609) [ 2217.453638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2217.462082] Lustre: *** cfs_fail_loc=197, val=0*** [ 2235.171203] LustreError: 103022:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 1: rc = -5 [ 2246.434123] LustreError: 103022:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 2252.608561] Lustre: *** cfs_fail_loc=197, val=0*** [ 2252.610551] Lustre: Skipped 65 previous similar messages [ 2275.231302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2275.241858] Lustre: Skipped 2 previous similar messages [ 2279.978350] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2287.578503] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2296.234671] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 09:48:16 (1761313696) [ 2314.852795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.148162] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.172790] LustreError: Skipped 6 previous similar messages [ 2318.196081] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2325.407168] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2328.829837] LustreError: 104681:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2328.836230] LustreError: 104681:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2328.851465] LustreError: 104681:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 2335.199142] Lustre: 105371:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313730/real 1761313730] req@00000000f5f94f8a x1846870313945792/t0(0) o251->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761313736 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2350.048209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2366.324367] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 2373.892592] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2385.683363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2391.902951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2398.726756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2405.162190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2412.391661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2412.493777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.784526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2412.808982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2412.859851] Lustre: lustre-MDT0000: new disk, initializing [ 2412.962410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2417.319741] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2427.597313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2427.687912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.038538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2432.129569] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2441.088796] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2446.420434] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2452.348814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2452.395195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2452.399924] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2456.036957] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2467.158273] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2472.211250] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2472.472039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2478.015776] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2478.351691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2485.739544] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2485.740629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2485.743098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2485.743109] Lustre: Skipped 11 previous similar messages [ 2485.750168] LustreError: Skipped 1 previous similar message [ 2485.805056] Lustre: Skipped 5 previous similar messages [ 2487.001748] Lustre: server umount lustre-OST0000 complete [ 2487.003498] Lustre: Skipped 3 previous similar messages [ 2496.727037] LustreError: 112686:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (0000000025404882) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2500.356686] LustreError: 108997:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313902 with bad export cookie 13590857697725283317 [ 2500.358858] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2500.367652] LustreError: 108997:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2507.846208] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 09:51:47 (1761313907) [ 2516.707730] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2519.342555] Key type lgssc unregistered [ 2519.618029] LNet: 113685:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2519.631933] LNet: Removed LNI 192.168.202.159@tcp [ 2520.318327] Key type .llcrypt unregistered [ 2520.320579] Key type ._llcrypt unregistered [ 2536.222743] alg: No test for adler32 (adler32-zlib) [ 2536.989768] Key type ._llcrypt registered [ 2536.991371] Key type .llcrypt registered [ 2537.120799] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 2548.697318] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2549.555045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2549.850168] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2550.043380] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2550.048253] LNet: Accept secure, port 988 [ 2551.712059] Key type lgssc registered [ 2552.604175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2560.003111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2565.514299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2571.127452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2575.978450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2581.261060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2587.416520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2587.506145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.892822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2588.946438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2589.073888] Lustre: lustre-MDT0000: new disk, initializing [ 2589.192884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.218056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2593.537934] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2603.228698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2603.306170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2603.315632] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2603.680728] Lustre: lustre-OST0000: new disk, initializing [ 2603.698447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2603.708642] Lustre: Skipped 1 previous similar message [ 2603.824785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2607.985481] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2617.940424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2618.050248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2618.057326] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2618.202087] Lustre: lustre-OST0001: new disk, initializing [ 2618.205714] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2618.288854] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2622.473726] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2634.721145] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2634.729582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2634.735127] Lustre: Skipped 1 previous similar message [ 2636.542893] Lustre: server umount lustre-MDT0000 complete [ 2642.657071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2646.527300] LustreError: 117172:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314048 with bad export cookie 12128125481200902650 [ 2646.532152] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.546246] LustreError: 117172:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2646.608117] Lustre: server umount lustre-OST0000 complete [ 2649.755310] Lustre: server umount lustre-OST0001 complete [ 2657.997674] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 09:54:17 (1761314057) [ 2664.943788] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2672.145881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2673.173110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2678.256093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2679.402638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2685.977990] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 09:54:45 (1761314085) [ 2698.993080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2701.001906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2702.879163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2704.701305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2705.897223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2707.811335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2723.381100] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 2730.949740] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2740.537894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2747.006290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2753.794801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2760.380579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2767.425842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2767.506588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.728851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2767.768322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2767.879854] Lustre: lustre-MDT0000: new disk, initializing [ 2767.997668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.027746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2772.347572] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2783.540286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2783.657237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.781200] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2783.865570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2783.880578] Lustre: Skipped 1 previous similar message [ 2783.980870] Lustre: lustre-MDT0001: new disk, initializing [ 2784.097092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2784.132467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2784.136678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2788.304978] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2797.074886] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2802.183477] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2807.404842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2807.446802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2807.450588] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2807.575779] Lustre: lustre-OST0000: new disk, initializing [ 2807.581042] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2807.657262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2812.157120] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2815.021346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2815.041440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2822.031960] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2826.887963] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2827.112994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2831.550300] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2831.776900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2835.945346] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2835.954721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.969506] Lustre: Skipped 1 previous similar message [ 2835.980837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2835.994658] Lustre: Skipped 1 previous similar message [ 2839.869883] Lustre: server umount lustre-OST0000 complete [ 2845.664186] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2845.676850] LustreError: Skipped 1 previous similar message [ 2845.693960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2845.724120] Lustre: Skipped 1 previous similar message [ 2845.733162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.425838] Lustre: server umount lustre-MDT0000 complete [ 2851.301650] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.321432] LustreError: Skipped 1 previous similar message [ 2852.324614] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.331928] LustreError: Skipped 1 previous similar message [ 2854.506711] LustreError: 125709:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314256 with bad export cookie 12128125481200903861 [ 2854.516274] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2854.603100] LustreError: 125709:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2865.403170] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 09:57:44 (1761314264) [ 2872.647100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2873.173334] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.274739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.141398] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2878.456497] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.557583] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.050869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.064824] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2901.032580] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2907.295884] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2914.688160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2915.074543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.083928] Lustre: Skipped 1 previous similar message [ 2919.062887] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 2926.549802] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2930.749582] Lustre: *** cfs_fail_loc=159, val=0*** [ 2930.751335] Lustre: Skipped 1 previous similar message [ 2937.218588] Lustre: lustre-MDT0000: Client 82c0f538-c457-4e1b-8ad1-261e7c955abf (at 192.168.202.59@tcp) reconnecting [ 2938.113043] Lustre: *** cfs_fail_loc=159, val=0*** [ 2938.118760] Lustre: Skipped 5 previous similar messages [ 2945.334895] Lustre: lustre-MDT0000: Client 82c0f538-c457-4e1b-8ad1-261e7c955abf (at 192.168.202.59@tcp) reconnecting [ 2950.115704] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2950.119392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.124494] Lustre: Skipped 1 previous similar message [ 2950.127070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.131641] Lustre: Skipped 2 previous similar messages [ 2955.245393] Lustre: server umount lustre-OST0000 complete [ 2955.253623] Lustre: Skipped 1 previous similar message [ 2960.357981] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2960.361712] LustreError: Skipped 1 previous similar message [ 2960.372666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.385751] Lustre: Skipped 1 previous similar message [ 2960.397571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.400302] Lustre: Skipped 2 previous similar messages [ 2966.503150] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.519866] LustreError: Skipped 1 previous similar message [ 2969.047687] LustreError: 130254:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314370 with bad export cookie 12128125481200904715 [ 2969.050262] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2969.062217] LustreError: 130254:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2979.786587] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2982.120199] Key type lgssc unregistered [ 2982.488761] LNet: 134123:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2982.508904] LNet: Removed LNI 192.168.202.159@tcp [ 2983.392673] Key type .llcrypt unregistered [ 2983.394443] Key type ._llcrypt unregistered [ 3001.786532] alg: No test for adler32 (adler32-zlib) [ 3002.548498] Key type ._llcrypt registered [ 3002.549809] Key type .llcrypt registered [ 3002.652378] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3003.520479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3004.135728] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 3004.369800] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3004.377490] LNet: Accept secure, port 988 [ 3006.063146] Key type lgssc registered [ 3007.185819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3014.721073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3016.095886] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3016.152642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3020.035675] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3021.280629] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.404155] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3028.776664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.107509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3033.551990] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3040.254838] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3045.223405] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3051.354711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3051.669489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3055.570764] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3063.504548] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3065.996938] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3077.266616] Lustre: *** cfs_fail_loc=159, val=0*** [ 3084.600025] Lustre: lustre-MDT0000: Client 5988e168-2f0f-4e65-9366-a1422bfb7c4f (at 192.168.202.59@tcp) reconnecting [ 3086.119368] Lustre: *** cfs_fail_loc=159, val=0*** [ 3093.334677] Lustre: lustre-MDT0001: Client 5988e168-2f0f-4e65-9366-a1422bfb7c4f (at 192.168.202.59@tcp) reconnecting [ 3094.158544] Lustre: *** cfs_fail_loc=159, val=0*** [ 3094.161188] Lustre: Skipped 5 previous similar messages [ 3101.547525] Lustre: lustre-MDT0001: Client 5988e168-2f0f-4e65-9366-a1422bfb7c4f (at 192.168.202.59@tcp) reconnecting [ 3107.324335] Lustre: *** cfs_fail_loc=159, val=0*** [ 3107.326354] Lustre: Skipped 2 previous similar messages [ 3114.800415] Lustre: lustre-MDT0000: Client 00c7f8f8-c4e4-452a-a96a-4d0dcc0dc00c (at 192.168.202.59@tcp) reconnecting [ 3115.870439] Lustre: *** cfs_fail_loc=159, val=0*** [ 3115.872349] Lustre: Skipped 13 previous similar messages [ 3123.084787] Lustre: lustre-MDT0000: Client 00c7f8f8-c4e4-452a-a96a-4d0dcc0dc00c (at 192.168.202.59@tcp) reconnecting [ 3129.837448] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3129.837518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.844656] LustreError: Skipped 2 previous similar messages [ 3129.874432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.855273] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3130.861766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.871726] LustreError: Skipped 4 previous similar messages [ 3130.903497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3132.905649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3132.914433] Lustre: Skipped 1 previous similar message [ 3138.029316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3138.040790] Lustre: Skipped 1 previous similar message [ 3143.137284] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3143.142323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3143.158152] Lustre: Skipped 1 previous similar message [ 3143.347977] Lustre: server umount lustre-OST0000 complete [ 3146.723372] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3146.731191] LustreError: Skipped 4 previous similar messages [ 3146.733594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.833745] Lustre: server umount lustre-MDT0000 complete [ 3153.380989] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.387761] LustreError: Skipped 1 previous similar message [ 3156.636054] LustreError: 135568:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314558 with bad export cookie 5769479116025225987 [ 3156.644618] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3156.652881] LustreError: 135568:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3157.065488] Lustre: server umount lustre-MDT0001 complete [ 3171.804950] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3175.396399] Key type lgssc unregistered [ 3175.620873] LNet: 139993:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3175.634474] LNet: Removed LNI 192.168.202.159@tcp [ 3176.316885] Key type .llcrypt unregistered [ 3176.319941] Key type ._llcrypt unregistered [ 3197.487416] alg: No test for adler32 (adler32-zlib) [ 3198.246978] Key type ._llcrypt registered [ 3198.250538] Key type .llcrypt registered [ 3198.359252] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3199.289575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3199.540727] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 3199.714361] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3199.717342] LNet: Accept secure, port 988 [ 3201.359099] Key type lgssc registered [ 3202.061735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3207.769261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.125129] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.164547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.681520] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3214.304821] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.275338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.425712] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.631405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3222.811827] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3229.854809] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3235.418826] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3241.205462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3241.523689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3245.700409] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3249.712180] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 3252.902902] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3258.959241] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:129 [ 3271.440394] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 3272.381945] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight in log params [ 3275.838478] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 3284.451099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3284.454818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3284.455405] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3284.508326] Lustre: Skipped 1 previous similar message [ 3287.456819] Lustre: server umount lustre-OST0000 complete [ 3291.615770] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3291.627285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.640332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3291.643080] Lustre: Skipped 1 previous similar message [ 3294.691160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.691497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.838643] Lustre: server umount lustre-MDT0000 complete [ 3299.809933] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3299.826922] LustreError: Skipped 1 previous similar message [ 3301.557888] LustreError: 144525:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314703 with bad export cookie 932944761038375812 [ 3301.562967] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3301.570218] LustreError: 144525:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3301.895335] Lustre: server umount lustre-MDT0001 complete [ 3313.151537] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3315.701620] Key type lgssc unregistered [ 3315.949737] LNet: 145535:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3315.973256] LNet: Removed LNI 192.168.202.159@tcp [ 3316.718777] Key type .llcrypt unregistered [ 3316.721572] Key type ._llcrypt unregistered [ 3338.141700] alg: No test for adler32 (adler32-zlib) [ 3338.898516] Key type ._llcrypt registered [ 3338.900818] Key type .llcrypt registered [ 3338.977515] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3339.885937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3340.177341] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 3340.316794] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3340.320330] LNet: Accept secure, port 988 [ 3341.944030] Key type lgssc registered [ 3342.739439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3350.242219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.534161] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3351.588967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3354.831939] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3356.641431] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3361.766189] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.227312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.703305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3367.028739] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3374.100697] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3379.507466] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3386.830311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3387.217741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.499774] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3393.323447] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 3398.611334] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3401.539467] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:161 [ 3408.684191] Lustre: *** cfs_fail_loc=159, val=0*** [ 3415.878116] Lustre: lustre-MDT0000: Client d232750f-4e89-4375-a482-d8a3cf728391 (at 192.168.202.59@tcp) reconnecting [ 3422.185133] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3422.186387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.193959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3422.202632] Lustre: Skipped 1 previous similar message [ 3425.879470] Lustre: server umount lustre-OST0000 complete [ 3429.346982] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3429.354551] LustreError: Skipped 4 previous similar messages [ 3429.357729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.365266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3429.367850] Lustre: Skipped 1 previous similar message [ 3432.418216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3432.421233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3432.443677] Lustre: Skipped 1 previous similar message [ 3435.342138] Lustre: server umount lustre-MDT0000 complete [ 3437.538357] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.019724] LustreError: 146982:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314840 with bad export cookie 8865492851505867282 [ 3439.042220] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3439.324191] Lustre: server umount lustre-MDT0001 complete [ 3449.468737] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3452.026219] Key type lgssc unregistered [ 3452.352648] LNet: 150763:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3452.364657] LNet: Removed LNI 192.168.202.159@tcp [ 3453.142173] Key type .llcrypt unregistered [ 3453.144114] Key type ._llcrypt unregistered [ 3473.337779] alg: No test for adler32 (adler32-zlib) [ 3474.104982] Key type ._llcrypt registered [ 3474.106880] Key type .llcrypt registered [ 3474.205161] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3475.488385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3475.960042] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 3476.238154] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3476.245134] LNet: Accept secure, port 988 [ 3477.951144] Key type lgssc registered [ 3479.011578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3486.644241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3488.175237] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3488.265311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.624457] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3493.345374] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.481939] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3501.213368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.597671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3505.554644] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3512.851775] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3518.414348] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3524.724528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3525.177654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3530.091696] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3532.344986] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 3537.097349] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3538.489604] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 3560.433855] Lustre: lustre-OST0000: evicting f08715e4-e003-4e3f-bf0c-a83e3dafe015 (at 192.168.202.59@tcp) by administrative request [ 3560.445747] Lustre: Skipped 1 previous similar message [ 3588.581757] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3588.585388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.594508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3589.601524] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3589.613486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.634408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3592.673134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3597.801038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3597.815317] Lustre: Skipped 2 previous similar messages [ 3602.399093] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3602.644992] Lustre: server umount lustre-OST0000 complete [ 3608.033671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.041042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3608.050866] Lustre: Skipped 1 previous similar message [ 3612.420049] Lustre: server umount lustre-MDT0000 complete [ 3613.154239] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3613.177214] LustreError: Skipped 1 previous similar message [ 3616.038092] LustreError: 155620:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315017 with bad export cookie 15894960729361103393 [ 3616.043403] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3616.063409] LustreError: 155620:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3616.554562] Lustre: server umount lustre-MDT0001 complete [ 3628.081362] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3631.061897] Key type lgssc unregistered [ 3631.302805] LNet: 156427:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3631.314538] LNet: Removed LNI 192.168.202.159@tcp [ 3632.290301] Key type .llcrypt unregistered [ 3632.294738] Key type ._llcrypt unregistered [ 3696.480349] alg: No test for adler32 (adler32-zlib) [ 3697.240546] Key type ._llcrypt registered [ 3697.247283] Key type .llcrypt registered [ 3697.365932] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3698.553742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3699.029331] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 3699.227581] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3699.234576] LNet: Accept secure, port 988 [ 3700.919128] Key type lgssc registered [ 3702.054581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3709.680351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.233625] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3711.337703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3715.824325] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3716.578097] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.707718] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3723.608854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.066556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3727.804918] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3735.090903] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3740.362308] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3746.115361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3746.412317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3750.527661] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3753.584415] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 3758.588484] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3761.779081] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 3772.384834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.417644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3776.219578] Lustre: server umount lustre-OST0000 complete [ 3780.576364] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3780.586189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.616238] Lustre: Skipped 1 previous similar message [ 3780.627366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3780.635411] Lustre: Skipped 1 previous similar message [ 3782.624022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3782.641435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3785.918296] Lustre: server umount lustre-MDT0000 complete [ 3787.749891] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.759053] LustreError: Skipped 1 previous similar message [ 3789.827708] LustreError: 158948:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315191 with bad export cookie 5007692840504009308 [ 3789.829029] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3789.839945] LustreError: 158948:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3790.202339] Lustre: server umount lustre-MDT0001 complete [ 3800.094250] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3802.872941] Key type lgssc unregistered [ 3803.188131] LNet: 162630:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3803.216563] LNet: Removed LNI 192.168.202.159@tcp [ 3804.138468] Key type .llcrypt unregistered [ 3804.141809] Key type ._llcrypt unregistered [ 3827.885917] alg: No test for adler32 (adler32-zlib) [ 3828.671197] Key type ._llcrypt registered [ 3828.673326] Key type .llcrypt registered [ 3828.836665] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3830.086301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3830.860490] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 3831.095612] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3831.100812] LNet: Accept secure, port 988 [ 3832.863391] Key type lgssc registered [ 3834.025621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3840.625281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3849.242588] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 10:14:08 (1761315248) [ 3865.268315] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3873.460500] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3886.509366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3894.267956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3900.780048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3906.405629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3912.966368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3913.091687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.274436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3914.296941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3914.350883] Lustre: lustre-MDT0000: new disk, initializing [ 3914.463437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3914.516191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3919.365692] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3931.397182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3931.497991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.677890] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3931.752832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3931.764460] Lustre: Skipped 1 previous similar message [ 3931.916438] Lustre: lustre-MDT0001: new disk, initializing [ 3932.063374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3932.104910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3932.121813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3936.418640] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3946.402776] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3952.781799] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3959.553567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3959.613351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3959.629513] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3959.968389] Lustre: lustre-OST0000: new disk, initializing [ 3959.978492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3960.098660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.355444] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 3969.110684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3969.114515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3973.980950] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3984.869207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.872062] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3984.873479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.879233] Lustre: Skipped 1 previous similar message [ 3989.985276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3989.994163] Lustre: Skipped 2 previous similar messages [ 3995.105813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3995.110836] Lustre: Skipped 1 previous similar message [ 3996.127147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3996.363657] Lustre: server umount lustre-OST0000 complete [ 4000.227269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.232085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4000.236992] Lustre: Skipped 1 previous similar message [ 4005.349693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4005.365770] Lustre: Skipped 2 previous similar messages [ 4006.030590] Lustre: server umount lustre-MDT0000 complete [ 4009.366195] LustreError: 167033:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315411 with bad export cookie 8970660251186155797 [ 4009.367123] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.375846] LustreError: 167033:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4009.741856] Lustre: server umount lustre-MDT0001 complete [ 4018.964624] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4021.409303] Key type lgssc unregistered [ 4021.659627] LNet: 171042:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4021.673409] LNet: Removed LNI 192.168.202.159@tcp [ 4022.488169] Key type .llcrypt unregistered [ 4022.492250] Key type ._llcrypt unregistered [ 4043.650185] alg: No test for adler32 (adler32-zlib) [ 4044.408517] Key type ._llcrypt registered [ 4044.425054] Key type .llcrypt registered [ 4044.504061] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4045.245034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4045.625064] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 4045.842989] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4045.846735] LNet: Accept secure, port 988 [ 4047.530593] Key type lgssc registered [ 4048.611748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4056.730699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4058.432367] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4058.492595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4063.485717] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4063.713413] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.832985] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4072.625149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4072.960627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4077.940924] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4085.568473] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4090.118417] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4095.407573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4095.647048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4099.094057] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4106.222212] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4108.975680] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4115.242291] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4115.254486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.262887] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4115.265887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4115.278021] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.159@tcp (at 0@lo) [ 4115.282399] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 4116.450482] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4116.462523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.473346] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4116.483323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4116.515048] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.159@tcp (at 0@lo) [ 4116.521945] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 4117.632568] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4117.643398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4117.653850] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4117.659764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4117.664842] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.159@tcp (at 0@lo) [ 4117.665425] Lustre: lustre-OST0000: deleting orphan objects from 0x0:110 to 0x0:129 [ 4118.471693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4118.480487] Lustre: lustre-OST0000: deleting orphan objects from 0x0:110 to 0x0:161 [ 4119.244184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4119.271905] Lustre: lustre-OST0000: deleting orphan objects from 0x0:110 to 0x0:193 [ 4120.539332] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4120.546992] Lustre: Skipped 2 previous similar messages [ 4120.565979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.582405] Lustre: Skipped 2 previous similar messages [ 4120.600694] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4120.615119] Lustre: Skipped 2 previous similar messages [ 4120.624359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4120.640226] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.159@tcp (at 0@lo) [ 4120.646944] Lustre: lustre-OST0000: deleting orphan objects from 0x0:110 to 0x0:225 [ 4120.666401] Lustre: Skipped 2 previous similar messages [ 4121.439800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4121.459484] Lustre: lustre-OST0000: deleting orphan objects from 0x0:110 to 0x0:257 [ 4122.416604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4122.427659] Lustre: lustre-OST0000: deleting orphan objects from 0x0:110 to 0x0:289 [ 4123.210636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4123.219299] Lustre: lustre-OST0000: deleting orphan objects from 0x0:110 to 0x0:321 [ 4124.097251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4124.113374] Lustre: lustre-OST0000: deleting orphan objects from 0x0:110 to 0x0:353 [ 4133.858335] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4133.869779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4133.880056] Lustre: Skipped 4 previous similar messages [ 4133.885704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4134.368509] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4134.394151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4138.612863] Lustre: server umount lustre-OST0000 complete [ 4144.608145] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4144.611755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.633280] Lustre: Skipped 1 previous similar message [ 4144.638196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.640778] Lustre: Skipped 1 previous similar message [ 4148.554551] Lustre: server umount lustre-MDT0000 complete [ 4150.241906] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.252533] LustreError: Skipped 1 previous similar message [ 4152.266932] LustreError: 172487:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315553 with bad export cookie 13526980515401812359 [ 4152.269166] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4152.276618] LustreError: 172487:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4152.636625] Lustre: server umount lustre-MDT0001 complete [ 4163.544898] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4166.572753] Key type lgssc unregistered [ 4166.852206] LNet: 176968:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4166.861555] LNet: Removed LNI 192.168.202.159@tcp [ 4167.580535] Key type .llcrypt unregistered [ 4167.582493] Key type ._llcrypt unregistered [ 4188.541837] alg: No test for adler32 (adler32-zlib) [ 4189.298625] Key type ._llcrypt registered [ 4189.306278] Key type .llcrypt registered [ 4189.462899] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4190.446800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4190.797771] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 4190.964666] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4190.970543] LNet: Accept secure, port 988 [ 4192.639153] Key type lgssc registered [ 4193.819313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4201.542984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4202.880516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.931209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.739589] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4208.101891] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4213.217572] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4214.618530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4214.914127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4219.285788] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4226.951865] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4233.332870] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4239.920312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4240.357093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4244.440878] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4251.899906] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4253.689415] Lustre: lustre-OST0000: deleting orphan objects from 0x0:110 to 0x0:385 [ 4260.345231] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4261.740265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.747270] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4261.753806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4261.761924] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.159@tcp (at 0@lo) [ 4261.761945] LustreError: 178451:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 4263.906683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.920169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4263.930060] Lustre: Skipped 1 previous similar message [ 4263.941245] Lustre: Skipped 1 previous similar message [ 4267.822919] Lustre: lustre-OST0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 4269.026257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4269.034388] Lustre: Skipped 1 previous similar message [ 4269.537124] Lustre: server umount lustre-OST0000 complete [ 4281.800787] LustreError: 178451:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 4281.806191] LustreError: 178451:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 4302.707553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4303.036788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4307.103850] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4311.206442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4311.216709] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4311.216729] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.159@tcp (at 0@lo) [ 4311.233975] Lustre: lustre-OST0000: deleting orphan objects from 0x0:110 to 0x0:417 [ 4311.235466] Lustre: Skipped 1 previous similar message [ 4311.244424] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:66 to 0x280000400:97 [ 4314.091328] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4322.990078] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 4328.066895] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9918c8805800.ost_server_uuid 40 [ 4329.445780] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9918c8805800.ost_server_uuid in FULL state after 0 sec [ 4337.120931] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4337.126131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.135568] LustreError: Skipped 1 previous similar message [ 4337.136859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4337.164735] Lustre: Skipped 1 previous similar message [ 4339.065744] Lustre: server umount lustre-OST0000 complete [ 4342.754316] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4342.759332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.776561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4342.780150] Lustre: Skipped 1 previous similar message [ 4347.362884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.606821] Lustre: server umount lustre-MDT0000 complete [ 4352.051387] LustreError: 178412:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315753 with bad export cookie 4852106977768920810 [ 4352.054679] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4352.060116] LustreError: 178412:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4352.565466] Lustre: server umount lustre-MDT0001 complete [ 4361.873793] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4364.115588] Key type lgssc unregistered [ 4364.374972] LNet: 183727:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4364.380939] LNet: Removed LNI 192.168.202.159@tcp [ 4365.014710] Key type .llcrypt unregistered [ 4365.019438] Key type ._llcrypt unregistered [ 4391.747750] alg: No test for adler32 (adler32-zlib) [ 4392.502989] Key type ._llcrypt registered [ 4392.504635] Key type .llcrypt registered [ 4392.593822] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4393.862159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4394.230814] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 4394.453077] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4394.460378] LNet: Accept secure, port 988 [ 4396.127096] Key type lgssc registered [ 4397.296282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4406.296508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.793866] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.845264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4412.483318] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4412.899250] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4418.024225] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4420.689750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.112537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4424.630611] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4432.523510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4432.921909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.294588] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4438.077849] Lustre: lustre-OST0000: deleting orphan objects from 0x0:419 to 0x0:449 [ 4445.429511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4445.501180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4445.511586] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4445.620547] Lustre: lustre-OST0001: new disk, initializing [ 4445.623339] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4445.711467] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4449.552850] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4453.961928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4453.967436] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4459.598574] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4466.536565] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4485.095744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.101101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4485.103763] Lustre: Skipped 3 previous similar messages [ 4485.109455] Lustre: Skipped 3 previous similar messages [ 4490.211022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4490.214982] Lustre: Skipped 3 previous similar messages [ 4495.331566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4495.345440] Lustre: Skipped 3 previous similar messages [ 4498.399330] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4498.544867] Lustre: server umount lustre-MDT0000 complete [ 4500.454472] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.475408] LustreError: Skipped 3 previous similar messages [ 4502.433849] LustreError: 185487:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315904 with bad export cookie 14593212171811529461 [ 4502.442269] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4502.455735] LustreError: 185487:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4502.800195] Lustre: server umount lustre-MDT0001 complete [ 4512.671144] Lustre: 184768:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315907/real 1761315907] req@0000000094ab5bc2 x1846873472152704/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315914 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 4512.704384] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.807050] Lustre: server umount lustre-OST0000 complete [ 4514.789402] Lustre: 184768:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315909/real 1761315909] req@000000009500149e x1846873472152896/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315916 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 4514.836853] Lustre: 184768:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4516.719789] Lustre: server umount lustre-OST0001 complete [ 4519.323550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4519.396470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4520.606291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4520.735754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4521.927568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4522.012292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4522.990505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4523.100817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4536.398315] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4545.248670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.618541] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.733707] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4549.650755] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4556.262708] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4556.277952] LustreError: Skipped 1 previous similar message [ 4557.351141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.651242] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4561.890523] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4569.613769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4573.031419] LustreError: 137-5: mylustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.720772] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4576.119067] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:452 to 0x0:481 [ 4582.162771] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4582.375510] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4582.382361] Lustre: Skipped 1 previous similar message [ 4586.679712] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4594.623842] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4598.121727] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4613.093576] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4613.096354] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4613.104393] Lustre: Skipped 2 previous similar messages [ 4613.111093] Lustre: Skipped 3 previous similar messages [ 4618.212745] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4618.221133] Lustre: Skipped 3 previous similar messages [ 4618.323278] Lustre: server umount mylustre-MDT0000 complete [ 4621.613703] LustreError: 190937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316023 with bad export cookie 14593212171811532184 [ 4621.619677] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4621.627275] LustreError: 190937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4630.431188] Lustre: 184769:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316025/real 1761316025] req@00000000f880085d x1846873472183360/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761316032 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 4630.457501] Lustre: 184769:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4630.466661] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4630.488366] Lustre: Skipped 1 previous similar message [ 4632.103209] Lustre: server umount mylustre-OST0000 complete [ 4632.108472] Lustre: Skipped 1 previous similar message [ 4633.567207] Lustre: 184771:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316028/real 1761316028] req@000000001eac6091 x1846873472183552/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761316035 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 4633.583183] Lustre: 184771:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4638.111379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4638.236570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4639.336627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4639.462282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4640.472893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4640.581065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4641.562679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4641.661109] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4655.244717] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4664.830189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.488571] LustreError: 137-5: tfs-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.505962] LustreError: Skipped 4 previous similar messages [ 4665.579503] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.065932] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4678.704934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.060474] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4689.437872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4693.419685] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4695.920723] Lustre: tfs-OST0000: deleting orphan objects from 0x0:452 to 0x0:513 [ 4702.069606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4703.959964] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 4706.447954] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4713.718801] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4717.644260] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4732.899431] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4732.901255] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 4732.934131] Lustre: Skipped 2 previous similar messages [ 4732.985334] Lustre: Skipped 1 previous similar message [ 4733.260614] Lustre: server umount tfs-MDT0000 complete [ 4733.262371] Lustre: Skipped 1 previous similar message [ 4737.634056] LustreError: 196302:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316139 with bad export cookie 14593212171811533794 [ 4737.637708] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4737.642584] LustreError: 196302:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4745.055236] Lustre: 184770:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316139/real 1761316139] req@0000000088ee1de0 x1846873472208704/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761316146 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 4745.075706] Lustre: 184770:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4745.092199] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4745.097627] Lustre: Skipped 1 previous similar message [ 4755.784635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4755.890689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4756.898229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4757.000144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4758.280570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4758.363898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4759.341993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4759.483970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4773.390752] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4782.268874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4782.653506] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.662650] LustreError: Skipped 7 previous similar messages [ 4782.754728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4782.757255] Lustre: Skipped 3 previous similar messages [ 4786.434878] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4794.661490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4799.053666] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4807.909738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4812.235434] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4812.551411] Lustre: lustre-OST0000: deleting orphan objects from 0x0:452 to 0x0:545 [ 4820.630560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4822.602728] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 4824.345196] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4831.534363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4834.915387] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4846.942256] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 10:30:46 (1761316246) [ 4866.532105] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4866.540346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4866.568239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4866.580188] Lustre: Skipped 2 previous similar messages [ 4868.613767] Lustre: server umount lustre-MDT0000 complete [ 4868.644736] Lustre: Skipped 3 previous similar messages [ 4872.619217] LustreError: 203964:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316274 with bad export cookie 14593212171811535425 [ 4872.622611] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4872.662197] LustreError: 203964:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4904.751087] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4913.497302] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4924.796625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4932.381852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4939.026582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4945.650069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4952.262441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4952.352632] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 4952.352632] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 4952.352632] [ 4952.368905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 4952.596072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4952.626202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4952.736550] Lustre: lustre-MDT0000: new disk, initializing [ 4952.907996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4952.916577] Lustre: Skipped 3 previous similar messages [ 4952.954662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4956.596165] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4966.079730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4966.125053] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 4966.125053] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 4966.125053] [ 4966.153342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 4966.267302] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4966.334899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4966.347264] Lustre: Skipped 1 previous similar message [ 4966.487100] Lustre: lustre-MDT0001: new disk, initializing [ 4966.710729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4966.731456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4970.940032] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4981.638678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4981.743345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4981.764311] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4982.097678] Lustre: lustre-OST0000: new disk, initializing [ 4982.103806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4983.998551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4984.033430] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4988.100281] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 4999.306191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4999.451437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4999.465874] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4999.673793] Lustre: lustre-OST0001: new disk, initializing [ 4999.680579] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5001.045416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5001.057680] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5005.135354] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5017.253542] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 10:33:36 (1761316416) [ 5031.142729] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 10:33:50 (1761316430) [ 5045.733139] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5045.737128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5045.754884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5045.761141] Lustre: Skipped 6 previous similar messages [ 5045.781309] Lustre: Skipped 7 previous similar messages [ 5051.674497] Lustre: server umount lustre-OST0000 complete [ 5051.676881] Lustre: Skipped 3 previous similar messages [ 5052.388659] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5052.399891] LustreError: Skipped 14 previous similar messages [ 5065.330598] LustreError: 208751:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316467 with bad export cookie 14593212171811536944 [ 5065.339952] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5065.352968] LustreError: 208751:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5081.699688] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5104.009315] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5117.554197] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5120.182611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5128.160614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5135.174456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5140.577517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5146.144818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5151.545918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5151.630864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5151.921457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5151.990908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5152.085297] Lustre: lustre-MDT0000: new disk, initializing [ 5152.194507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5155.974829] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5164.569863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5164.621960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.878040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5168.860184] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5177.112892] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5182.079897] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5187.254130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5187.316182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5187.320496] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5187.475797] Lustre: lustre-OST0000: new disk, initializing [ 5187.478300] Lustre: Skipped 1 previous similar message [ 5187.481739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5187.486635] Lustre: Skipped 2 previous similar messages [ 5189.482256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5189.493941] Lustre: Skipped 1 previous similar message [ 5189.496185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5190.502558] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5198.877991] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5204.964204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.965732] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5204.967165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5204.967170] Lustre: Skipped 7 previous similar messages [ 5204.973101] Lustre: Skipped 5 previous similar messages [ 5204.979455] LustreError: Skipped 1 previous similar message [ 5209.994855] Lustre: server umount lustre-OST0000 complete [ 5209.997280] Lustre: Skipped 3 previous similar messages [ 5222.445826] LustreError: 215967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316624 with bad export cookie 14593212171811538372 [ 5222.451614] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5222.456258] LustreError: 215967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5232.040472] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5234.175362] Key type lgssc unregistered [ 5234.380059] LNet: 218962:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5234.394026] LNet: Removed LNI 192.168.202.159@tcp [ 5234.981378] Key type .llcrypt unregistered [ 5234.982996] Key type ._llcrypt unregistered [ 5253.034401] alg: No test for adler32 (adler32-zlib) [ 5253.786582] Key type ._llcrypt registered [ 5253.789202] Key type .llcrypt registered [ 5253.849209] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5254.507643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5254.729360] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 5254.875890] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5254.887762] LNet: Accept secure, port 988 [ 5256.567124] Key type lgssc registered [ 5257.479098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5265.093187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5266.470831] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5266.545037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5270.209333] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5272.040760] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5277.161724] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5277.705560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5277.954213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5281.249994] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5288.388321] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5293.969426] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5300.339953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5300.593069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5304.470427] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5311.007605] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5311.984392] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 5315.730439] Lustre: server umount lustre-OST0000 complete [ 5319.136790] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5319.142105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5319.158655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5322.208110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5322.212804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5324.257066] Lustre: 219787:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316718/real 1761316718] req@00000000770f689d x1846874373895168/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761316725 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5324.292867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5324.302554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5324.310347] Lustre: Skipped 2 previous similar messages [ 5324.860263] Lustre: server umount lustre-MDT0000 complete [ 5325.280087] Lustre: 219785:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316719/real 1761316719] req@000000004c0cf969 x1846874373895296/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761316726 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 5328.355151] Lustre: 219786:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316723/real 1761316723] req@00000000249e5559 x1846874373896128/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761316730 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5328.907663] LustreError: 220455:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316730 with bad export cookie 8964571152183147270 [ 5328.909928] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5328.926074] LustreError: 220455:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5329.323225] Lustre: server umount lustre-MDT0001 complete [ 5340.995536] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5343.287723] Key type lgssc unregistered [ 5343.512844] LNet: 224178:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5343.517946] LNet: Removed LNI 192.168.202.159@tcp [ 5344.049846] Key type .llcrypt unregistered [ 5344.054340] Key type ._llcrypt unregistered [ 5356.094889] alg: No test for adler32 (adler32-zlib) [ 5356.850431] Key type ._llcrypt registered [ 5356.851928] Key type .llcrypt registered [ 5356.921541] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5357.492166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5357.921524] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 5358.049423] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5358.052972] LNet: Accept secure, port 988 [ 5359.663127] Key type lgssc registered [ 5360.550789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5367.307164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5384.863335] LustreError: 225420:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 1: rc = -5 [ 5415.583203] LustreError: 225420:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 5445.343181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5449.253850] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5456.570228] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5459.052842] Lustre: server umount lustre-OST0000 complete [ 5476.191357] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5478.602861] Key type lgssc unregistered [ 5478.901760] LNet: 227065:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5478.911737] LNet: Removed LNI 192.168.202.159@tcp [ 5479.674687] Key type .llcrypt unregistered [ 5479.681909] Key type ._llcrypt unregistered [ 5518.063274] alg: No test for adler32 (adler32-zlib) [ 5518.824861] Key type ._llcrypt registered [ 5518.826620] Key type .llcrypt registered [ 5518.959267] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5519.848072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5520.380206] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 5520.545116] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5520.557732] LNet: Accept secure, port 988 [ 5522.199123] Key type lgssc registered [ 5523.230948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5533.819047] loop0: detected capacity change from 0 to 204800000 [ 5533.976607] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5533.985963] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5534.315738] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5534.329298] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5534.379800] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5534.422329] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5534.455334] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5534.513158] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5534.561618] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5534.617729] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5535.537754] loop0: detected capacity change from 0 to 204800000 [ 5535.703323] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5536.356627] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5537.223981] loop0: detected capacity change from 0 to 204800000 [ 5537.630396] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5538.554304] loop0: detected capacity change from 0 to 204800000 [ 5538.915782] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5539.695627] loop0: detected capacity change from 0 to 204800000 [ 5539.718110] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5542.931289] loop0: detected capacity change from 0 to 204800000 [ 5542.975834] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5545.924744] loop0: detected capacity change from 0 to 204800000 [ 5545.940189] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5549.193696] loop0: detected capacity change from 0 to 204800000 [ 5549.213829] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5552.329263] loop0: detected capacity change from 0 to 204800000 [ 5552.450462] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5552.529471] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5552.593650] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5552.609381] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 5552.620483] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 5552.626897] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5552.634125] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5552.639825] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 5552.646247] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 5552.659717] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5552.672964] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 5552.689128] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 5552.717113] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 5552.737617] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 5552.761358] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 5552.775654] Lustre: 230546:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 5564.896478] Lustre: 230825:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316960/real 1761316960] req@000000000d3136f3 x1846874652803200/t0(0) o251->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761316966 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5564.967463] Lustre: server umount MGS complete [ 5565.998822] loop0: detected capacity change from 0 to 204800000 [ 5566.198550] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.706754] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5566.781512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5566.815556] LustreError: 230879:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 5568.039133] loop1: detected capacity change from 0 to 204800000 [ 5568.394662] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5568.544607] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5568.706661] Lustre: 230995:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5568.734328] Lustre: 230995:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 5568.751780] Lustre: 230995:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5568.778030] Lustre: 230995:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5568.809149] Lustre: 230995:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 5568.833088] Lustre: 230995:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5568.857488] Lustre: 230995:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 5568.906178] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5569.181070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5569.275733] LustreError: 230995:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0001: Aborting recovery for device [ 5570.536960] loop2: detected capacity change from 0 to 204800000 [ 5570.789490] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5570.880990] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5570.889981] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 5571.006166] Lustre: 231081:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5571.016231] Lustre: 231081:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5571.036142] Lustre: 231081:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5571.054986] Lustre: 231081:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 5571.073823] Lustre: 231081:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 5571.097133] Lustre: 231081:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5571.125904] Lustre: 231081:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5571.264606] Lustre: Found index 0 for lustre-OST0000, updating log [ 5571.433052] LustreError: 230937:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 5571.439914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5571.481498] LustreError: 231081:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 5572.456918] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5572.469763] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 5572.789768] loop3: detected capacity change from 0 to 204800000 [ 5572.922460] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5573.069054] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5573.081990] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 5573.142508] Lustre: 231169:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5573.153433] Lustre: 231169:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5573.166221] Lustre: 231169:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 5573.184816] Lustre: 231169:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5573.196588] Lustre: 231169:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5573.205349] Lustre: 231169:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 5573.221241] Lustre: 231169:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5573.252351] Lustre: Found index 1 for lustre-OST0001, updating log [ 5573.421215] LustreError: 231169:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 5574.458222] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 5574.461311] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 5574.467036] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 5581.856196] LustreError: 230901:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 952us: rc = -2 [ 5582.357328] LustreError: 231108:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 746us: rc = -2 [ 5582.367576] LustreError: 231108:0:(mdt_identity.c:112:mdt_identity_do_upcall()) Skipped 22 previous similar messages [ 5595.105494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5595.108810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5595.123860] Lustre: Skipped 2 previous similar messages [ 5595.124719] Lustre: Skipped 3 previous similar messages [ 5600.225371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5600.233336] Lustre: Skipped 4 previous similar messages [ 5601.216773] Lustre: server umount lustre-MDT0000 complete [ 5602.483371] LustreError: 230888:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317004 with bad export cookie 3685654887856090668 [ 5602.491319] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5602.504479] LustreError: 230888:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5605.344835] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.349526] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.370508] LustreError: Skipped 1 previous similar message [ 5605.399056] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5605.401744] Lustre: Skipped 1 previous similar message [ 5607.403567] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.404768] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5607.417052] LustreError: Skipped 1 previous similar message [ 5607.419367] Lustre: Skipped 1 previous similar message [ 5609.115110] Lustre: server umount lustre-MDT0001 complete [ 5616.413278] Lustre: server umount lustre-OST0000 complete [ 5623.700692] Lustre: server umount lustre-OST0001 complete [ 5631.356211] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 10:43:50 (1761317030) [ 5659.295149] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5666.465540] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5674.967705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5680.529686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5686.126213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5691.046272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5696.879672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5696.945599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.185965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5697.217686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5697.294543] Lustre: lustre-MDT0000: new disk, initializing [ 5697.364626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5697.373961] Lustre: Skipped 1 previous similar message [ 5697.399941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5701.047579] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5709.713766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5709.759923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.824927] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5709.857276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5709.860670] Lustre: Skipped 1 previous similar message [ 5709.925620] Lustre: lustre-MDT0001: new disk, initializing [ 5710.023443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5710.059600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5710.074215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5713.579476] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5721.978382] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5728.006595] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5733.622384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5733.670544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5733.674696] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5733.819938] Lustre: lustre-OST0000: new disk, initializing [ 5733.824767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5733.893710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.752925] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5742.117977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5742.128280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5745.411110] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5749.362288] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5753.360063] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5761.361626] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 5770.002424] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 5809.123728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.126493] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5809.127582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5809.127587] Lustre: Skipped 2 previous similar messages [ 5809.144610] Lustre: Skipped 2 previous similar messages [ 5810.570473] Lustre: server umount lustre-OST0000 complete [ 5814.241437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5814.250862] Lustre: Skipped 1 previous similar message [ 5819.360428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5819.363385] Lustre: Skipped 4 previous similar messages [ 5823.600284] LustreError: 238489:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317225 with bad export cookie 3685654887856093069 [ 5823.601978] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5823.631168] LustreError: 238489:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5831.279479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5835.027621] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5845.471167] Lustre: 240688:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317241/real 1761317241] req@0000000064b991d0 x1846874652870784/t0(0) o251->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761317247 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5845.530396] Lustre: server umount MGS complete [ 5845.532166] Lustre: Skipped 2 previous similar messages [ 5854.941373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.249663] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.262356] LustreError: Skipped 2 previous similar messages [ 5855.265484] LustreError: 241324:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 5855.330579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5858.803868] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5865.445073] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.455698] LustreError: Skipped 1 previous similar message [ 5865.485626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.114225] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5875.258736] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5880.323891] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5886.613683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5890.403941] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 5896.371641] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5932.007258] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5932.012389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5932.015745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5932.024828] Lustre: Skipped 1 previous similar message [ 5932.030256] Lustre: Skipped 1 previous similar message [ 5936.564241] Lustre: server umount lustre-OST0000 complete [ 5942.244487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5942.262172] Lustre: Skipped 1 previous similar message [ 5947.361475] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5947.369013] LustreError: Skipped 1 previous similar message [ 5951.016935] LustreError: 244128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317352 with bad export cookie 3685654887856094882 [ 5951.023246] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5951.031650] LustreError: 244128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5964.276578] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5966.963262] Key type lgssc unregistered [ 5967.313884] LNet: 245450:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5967.327265] LNet: Removed LNI 192.168.202.159@tcp [ 5968.100842] Key type .llcrypt unregistered [ 5968.102398] Key type ._llcrypt unregistered [ 5999.991211] alg: No test for adler32 (adler32-zlib) [ 6000.748577] Key type ._llcrypt registered [ 6000.750083] Key type .llcrypt registered [ 6000.849212] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 6012.963905] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6014.171211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6014.670710] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 6014.960632] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6014.971576] LNet: Accept secure, port 988 [ 6016.695163] Key type lgssc registered [ 6017.826554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6026.908388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6033.654270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6039.929636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6046.768246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6052.611861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6052.659248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6053.866127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6053.910069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6054.028547] Lustre: lustre-MDT0000: new disk, initializing [ 6054.139911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6054.168466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6058.881857] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6067.632712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6067.696417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6067.755373] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6067.796177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6067.799554] Lustre: Skipped 1 previous similar message [ 6067.849801] Lustre: lustre-MDT0001: new disk, initializing [ 6067.923968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6067.953504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6067.962651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6070.806876] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6077.295399] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6081.185524] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6086.610980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6086.686322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6086.693875] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6086.892382] Lustre: lustre-OST0000: new disk, initializing [ 6086.895389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6086.973836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6089.938222] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6094.887816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6094.893892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6096.850081] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6102.110873] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6106.442751] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6111.619961] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 6151.137923] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6151.141558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6151.154345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6151.653705] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6151.658272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6151.673346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6156.072542] Lustre: server umount lustre-OST0000 complete [ 6160.354859] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6160.358271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.365754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6165.211682] Lustre: server umount lustre-MDT0000 complete [ 6167.011672] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6167.021576] LustreError: Skipped 1 previous similar message [ 6168.034779] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.042542] LustreError: Skipped 1 previous similar message [ 6168.655231] LustreError: 249527:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317570 with bad export cookie 9292057337553915742 [ 6168.656904] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6168.688784] LustreError: 249527:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6169.090209] Lustre: server umount lustre-MDT0001 complete [ 6175.499456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.732380] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6189.025621] Lustre: 254775:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317584/real 1761317584] req@00000000248cd0f1 x1846875170840448/t0(0) o251->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761317590 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6189.075710] Lustre: server umount MGS complete [ 6199.481703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6199.887675] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6199.908238] LustreError: 255410:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 6199.966155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6204.064570] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6205.414291] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6210.537474] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6213.240206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6213.810946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6218.114491] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6226.067354] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6231.997400] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6240.027831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6240.370681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6244.333629] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6252.385834] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6284.772378] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6284.779393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6284.791141] Lustre: Skipped 1 previous similar message [ 6284.808065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6284.811459] Lustre: Skipped 2 previous similar messages [ 6289.355559] Lustre: server umount lustre-OST0000 complete [ 6295.520083] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6295.527377] LustreError: Skipped 1 previous similar message [ 6295.533827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.541451] Lustre: Skipped 1 previous similar message [ 6295.546854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6295.549799] Lustre: Skipped 2 previous similar messages [ 6298.811290] Lustre: server umount lustre-MDT0000 complete [ 6302.178810] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6302.200573] LustreError: Skipped 1 previous similar message [ 6302.334645] LustreError: 258193:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317704 with bad export cookie 9292057337553917429 [ 6302.336981] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6302.345388] LustreError: 258193:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6312.624682] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 6314.924584] Key type lgssc unregistered [ 6315.164681] LNet: 259515:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6315.175816] LNet: Removed LNI 192.168.202.159@tcp [ 6315.967986] Key type .llcrypt unregistered [ 6315.969468] Key type ._llcrypt unregistered [ 6338.950113] alg: No test for adler32 (adler32-zlib) [ 6339.718472] Key type ._llcrypt registered [ 6339.729920] Key type .llcrypt registered [ 6339.863515] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6340.786964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6341.084279] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 6341.244845] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6341.250024] LNet: Accept secure, port 988 [ 6342.879597] Key type lgssc registered [ 6343.889871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6351.220906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6352.615772] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6352.685128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6356.873862] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6357.985927] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6363.104689] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6364.335601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6364.669031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6368.383425] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6374.688719] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6378.872702] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6385.490109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6385.811030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6390.200722] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6397.450504] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6404.358392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6404.454066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6404.463894] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6404.588742] Lustre: lustre-OST0001: new disk, initializing [ 6404.594660] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6404.695951] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6408.732487] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6410.837752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6410.852389] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6410.903109] LustreError: 264408:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6411.937197] LustreError: 264403:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6411.959229] LustreError: 264403:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 6412.964589] LustreError: 264408:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6415.011754] LustreError: 264408:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6415.026485] LustreError: 264408:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 6417.869899] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6418.044743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6419.174504] LustreError: 264403:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6419.194389] LustreError: 264403:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [ 6423.263640] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9918d1396000.ost_server_uuid 40 [ 6424.864622] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9918d1396000.ost_server_uuid in FULL state after 0 sec [ 6427.424683] LustreError: 264403:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6427.432814] LustreError: 264403:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [ 6430.114541] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9918d1396000.ost_server_uuid 40 [ 6431.365811] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9918d1396000.ost_server_uuid in FULL state after 0 sec [ 6447.084024] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6447.091970] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6447.101626] Lustre: Skipped 1 previous similar message [ 6449.449467] Lustre: lustre-OST0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 6451.676552] Lustre: server umount lustre-OST0001 complete [ 6452.200205] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6452.211935] LustreError: Skipped 1 previous similar message [ 6456.805448] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6456.810530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6456.824015] Lustre: Skipped 1 previous similar message [ 6456.830119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6457.315223] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.331628] LustreError: Skipped 1 previous similar message [ 6462.093359] Lustre: server umount lustre-OST0000 complete [ 6467.041878] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6467.048385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.056323] Lustre: Skipped 1 previous similar message [ 6467.060197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6467.069200] Lustre: Skipped 2 previous similar messages [ 6471.529771] Lustre: server umount lustre-MDT0000 complete [ 6472.677572] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6474.526903] LustreError: 263996:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317876 with bad export cookie 11208202516446716283 [ 6474.532713] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6474.539294] LustreError: 263996:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6474.968565] Lustre: server umount lustre-MDT0001 complete [ 6484.186219] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 6486.589304] Key type lgssc unregistered [ 6486.833948] LNet: 266737:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6486.839315] LNet: Removed LNI 192.168.202.159@tcp [ 6487.526677] Key type .llcrypt unregistered [ 6487.533416] Key type ._llcrypt unregistered [ 6515.301513] alg: No test for adler32 (adler32-zlib) [ 6516.055547] Key type ._llcrypt registered [ 6516.057219] Key type .llcrypt registered [ 6516.144899] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6517.009595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6517.231746] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 6517.412748] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6517.418834] LNet: Accept secure, port 988 [ 6519.047362] Key type lgssc registered [ 6519.808573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6524.661742] loop0: detected capacity change from 0 to 1099511627776 [ 6524.979981] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 6524.988503] *** VALIDATE xfs *** [ 6524.997550] XFS (loop0): Mounting V5 Filesystem [ 6525.015453] XFS (loop0): Ending clean mount [ 6525.827506] loop1: detected capacity change from 0 to 18691697672192 [ 6557.583894] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6569.454853] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 6574.244493] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6580.403277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6584.812222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6588.818844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6593.131490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6598.100063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6598.140771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6599.297385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6599.316856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6599.363229] Lustre: lustre-MDT0000: new disk, initializing [ 6599.407218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.423519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6602.232101] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6609.847584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6609.896180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.973255] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6610.028767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6610.031705] Lustre: Skipped 1 previous similar message [ 6610.086329] Lustre: lustre-MDT0001: new disk, initializing [ 6610.131430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6610.152310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6610.156776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6612.767591] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6619.713467] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6623.729640] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6628.261041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6628.314339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6628.320043] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6628.489541] Lustre: lustre-OST0000: new disk, initializing [ 6628.497359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6628.568791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6632.126220] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6636.065080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6636.071807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6639.716760] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6643.165462] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6643.302646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6646.339496] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6646.501860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6651.364479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6651.366259] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6651.374305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6651.378679] Lustre: Skipped 1 previous similar message [ 6656.492830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6656.499256] Lustre: Skipped 1 previous similar message [ 6661.600755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6661.615668] Lustre: Skipped 1 previous similar message [ 6662.112618] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6662.317686] Lustre: server umount lustre-OST0000 complete [ 6666.722356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6666.723247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6666.727576] Lustre: Skipped 2 previous similar messages [ 6666.736907] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6666.742032] Lustre: Skipped 1 previous similar message [ 6671.784790] Lustre: server umount lustre-MDT0000 complete [ 6671.842304] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6671.870859] LustreError: Skipped 1 previous similar message [ 6674.821691] LustreError: 271673:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318076 with bad export cookie 6075981164742049231 [ 6674.826695] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6674.833475] LustreError: 271673:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6675.117764] Lustre: server umount lustre-MDT0001 complete [ 6681.052617] XFS (loop0): Unmounting Filesystem [ 6684.714283] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 11:01:24 (1761318084) [ 6690.003720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.383077] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6690.468131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6693.847349] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6695.904421] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6699.784349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6700.037253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6702.958940] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6708.090598] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6712.339931] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6717.940683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6718.277624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6721.763262] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6728.345475] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6735.840853] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6735.848774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6735.859423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6739.941558] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6739.944741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6745.057876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6745.063054] Lustre: Skipped 3 previous similar messages [ 6750.176483] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6750.310546] Lustre: server umount lustre-OST0000 complete [ 6755.297261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6755.303888] Lustre: Skipped 1 previous similar message [ 6759.429255] Lustre: server umount lustre-MDT0000 complete [ 6760.419918] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6760.434291] LustreError: Skipped 1 previous similar message [ 6762.108266] LustreError: 276369:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318163 with bad export cookie 6075981164742050085 [ 6762.112880] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6762.114491] LustreError: 276369:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6770.440482] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 6772.620453] Key type lgssc unregistered [ 6772.823491] LNet: 280138:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6772.835382] LNet: Removed LNI 192.168.202.159@tcp [ 6773.550096] Key type .llcrypt unregistered [ 6773.557565] Key type ._llcrypt unregistered [ 6791.983489] alg: No test for adler32 (adler32-zlib) [ 6792.737420] Key type ._llcrypt registered [ 6792.739607] Key type .llcrypt registered [ 6792.805868] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6793.368188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6793.585473] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 6793.689754] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6793.693217] LNet: Accept secure, port 988 [ 6795.335709] Key type lgssc registered [ 6795.949872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6801.924254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.212215] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6803.283542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6806.466415] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6808.547793] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6813.302129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6813.580444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6816.495863] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6821.971459] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6826.166098] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6830.707680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6830.922438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6834.466924] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6841.542455] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6853.951968] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 6854.624229] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6854.630774] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6854.646882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6854.667529] Lustre: Skipped 1 previous similar message [ 6854.697064] Lustre: Skipped 1 previous similar message [ 6857.652880] Lustre: server umount lustre-MDT0001 complete [ 6859.037697] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6860.160323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6864.150710] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6864.164595] LustreError: Skipped 2 previous similar messages [ 6864.842670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6864.932243] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6864.936893] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6864.970743] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6865.170864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6867.965937] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 6870.500708] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6870.508068] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6870.512592] Lustre: Skipped 1 previous similar message [ 6871.024413] LustreError: 285493:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 6871.030865] LustreError: 285493:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6871.036563] LustreError: 285493:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6871.041229] LustreError: 281744:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6871.076754] LustreError: 285496:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -17 [ 6871.084334] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 6871.084334] [ 7172.578613] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.585974] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7176.488160] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 7176.496940] Lustre: Skipped 1 previous similar message [ 7178.654419] Lustre: server umount lustre-MDT0001 complete [ 7180.415054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7181.591441] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7181.602409] LustreError: Skipped 2 previous similar messages [ 7184.616906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7184.674605] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7184.678367] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7184.703743] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7184.883477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7187.388800] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 7189.989932] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 7189.997147] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7192.566843] LustreError: 295850:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 7192.573976] LustreError: 295850:0:(genops.c:522:class_register_device()) Skipped 1 previous similar message [ 7192.578790] LustreError: 295850:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7192.583343] LustreError: 295850:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7192.588293] LustreError: 281744:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7192.593128] LustreError: 281744:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7192.636464] LustreError: 295862:0:(obd_config.c:680:class_setup()) Device 8 already setup (type osp) [ 7192.643213] LustreError: 295862:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -17 [ 7192.649724] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.159@tcp [ 7192.649724] [ 7492.067984] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7492.068511] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7492.072828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7492.076024] Lustre: Skipped 2 previous similar messages [ 7492.081941] Lustre: Skipped 3 previous similar messages [ 7502.305237] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7502.307941] Lustre: Skipped 5 previous similar messages [ 7505.375971] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7505.543912] Lustre: server umount lustre-MDT0001 complete [ 7507.301189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7507.430573] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7507.437199] LustreError: Skipped 3 previous similar messages [ 7512.056853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7512.128426] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7512.130851] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7512.153252] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7512.332073] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7514.642614] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 7517.666684] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7517.672510] LustreError: 306520:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 7517.676256] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 7517.676748] LustreError: 306520:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7517.676756] LustreError: 306520:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7517.677774] LustreError: 281744:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7517.683118] Lustre: Skipped 2 previous similar messages [ 7517.690559] LustreError: 281744:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7517.721286] LustreError: 306527:0:(lod_lov.c:228:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 7517.731153] LustreError: 306527:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -114 [ 7517.735785] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 7517.735785] [ 7829.986301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7829.986573] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7829.993851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7830.000618] Lustre: Skipped 3 previous similar messages [ 7835.607871] Lustre: server umount lustre-MDT0000 complete [ 7837.635111] LustreError: 281733:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319239 with bad export cookie 5305088326193861798 [ 7837.638068] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7837.639818] LustreError: 281733:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7840.224339] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7840.224531] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7840.234218] LustreError: Skipped 1 previous similar message [ 7840.239952] Lustre: Skipped 2 previous similar messages [ 7843.911678] Lustre: server umount lustre-MDT0001 complete [ 7850.164829] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 7854.146706] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7858.560521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7861.499053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7864.358966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7867.360347] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7870.347483] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 11:21:10 (1761319270) [ 7873.435436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7873.471508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7873.576173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7873.591104] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7873.632301] Lustre: lustre-MDT0000: new disk, initializing [ 7873.675092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7873.685848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7875.635863] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 7881.670772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7881.702929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7881.750675] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7881.778175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7881.781652] Lustre: Skipped 1 previous similar message [ 7881.826975] Lustre: lustre-MDT0001: new disk, initializing [ 7881.879293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7881.895223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7881.899985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7884.028365] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 7889.047962] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7891.980782] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7895.739104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7895.778275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7895.783064] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7895.914274] Lustre: lustre-OST0000: new disk, initializing [ 7895.917616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7895.975636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7898.402987] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all 8 [ 7902.226817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7902.232229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7904.123078] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7911.910389] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7911.916455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7911.925233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7911.928345] Lustre: Skipped 5 previous similar messages [ 7911.999408] Lustre: server umount lustre-MDT0000 complete [ 7912.001788] Lustre: Skipped 1 previous similar message [ 7912.417198] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7912.424356] LustreError: Skipped 3 previous similar messages [ 7913.981662] LustreError: 319820:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319315 with bad export cookie 5305088326193863709 [ 7913.981957] LustreError: 166-1: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7913.982548] LustreError: 320690:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 7913.982654] LustreError: 320690:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000026a4edde x1846875987818304/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' [ 7913.983118] LustreError: 320690:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 7913.983128] LustreError: 320690:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 7913.988585] LustreError: 319820:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7922.655371] Lustre: 323079:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319318/real 1761319318] req@00000000f7a9532e x1846875987818560/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761319324 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7937.991229] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 7942.065900] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7947.630331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7951.405260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7954.389181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7957.368954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7958.086401] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7728 sec ========== 11:22:38 (1761319358) [ 7975.544691] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7976.804118] Key type lgssc unregistered [ 7976.926581] LNet: 328176:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7976.930705] LNet: Removed LNI 192.168.202.159@tcp [ 7977.279539] Key type .llcrypt unregistered [ 7977.281170] Key type ._llcrypt unregistered