[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481120675 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.032 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002362] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007363] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4800.06 BogoMIPS (lpj=2400032) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.011095] LSM: Security Framework initializing [ 0.012062] Yama: becoming mindful. [ 0.013048] SELinux: Initializing. [ 0.014061] *** VALIDATE selinux *** [ 0.023013] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028751] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029166] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031111] *** VALIDATE tmpfs *** [ 0.032475] *** VALIDATE proc *** [ 0.033276] *** VALIDATE cgroup *** [ 0.034016] *** VALIDATE cgroup2 *** [ 0.035277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039015] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043182] debug: unmapping init [mem 0xffffffff87659000-0xffffffff87660fff] [ 0.045925] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051018] ... max period: 00007fffffffffff [ 0.052019] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.055236] rcu: Hierarchical SRCU implementation. [ 0.057561] smp: Bringing up secondary CPUs ... [ 0.058642] x86: Booting SMP configuration: [ 0.059033] .... node #0, CPUs: #1 #2 #3 [ 0.064668] smp: Brought up 1 node, 4 CPUs [ 0.066045] smpboot: Max logical packages: 1 [ 0.067023] smpboot: Total of 4 processors activated (19200.25 BogoMIPS) [ 0.235029] node 0 deferred pages initialised in 165ms [ 0.237416] devtmpfs: initialized [ 0.238238] x86/mm: Memory block size: 128MB [ 0.240763] gcov: version magic: 0x41383552 [ 0.242313] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.243090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.244267] pinctrl core: initialized pinctrl subsystem [ 0.245212] [ 0.245926] ************************************************************* [ 0.246020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247025] ** ** [ 0.248020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.249017] ** ** [ 0.250019] ** This means that this kernel is built to expose internal ** [ 0.251017] ** IOMMU data structures, which may compromise security on ** [ 0.252013] ** your system. ** [ 0.253015] ** ** [ 0.254016] ** If you see this message and you are not debugging the ** [ 0.255017] ** kernel, report this immediately to your vendor! ** [ 0.256019] ** ** [ 0.257018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258019] ************************************************************* [ 0.259519] NET: Registered protocol family 16 [ 0.260414] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.261062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.262061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.263454] cpuidle: using governor menu [ 0.266118] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.269739] PCI: Using configuration type 1 for base access [ 0.272151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.281120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.284044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.289041] cryptd: max_cpu_qlen set to 1000 [ 0.294241] ACPI: Added _OSI(Module Device) [ 0.296046] ACPI: Added _OSI(Processor Device) [ 0.298038] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.300022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.306868] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.314425] ACPI: Interpreter enabled [ 0.316103] ACPI: PM: (supports S0 S3 S4 S5) [ 0.318018] ACPI: Using IOAPIC for interrupt routing [ 0.320175] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.324449] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.334835] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.337066] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.340039] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.343109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.349390] acpiphp: Slot [2] registered [ 0.350454] acpiphp: Slot [5] registered [ 0.352178] acpiphp: Slot [6] registered [ 0.353167] acpiphp: Slot [7] registered [ 0.355169] acpiphp: Slot [8] registered [ 0.357173] acpiphp: Slot [9] registered [ 0.359176] acpiphp: Slot [10] registered [ 0.361156] acpiphp: Slot [3] registered [ 0.362135] acpiphp: Slot [4] registered [ 0.364151] acpiphp: Slot [11] registered [ 0.366146] acpiphp: Slot [12] registered [ 0.367143] acpiphp: Slot [13] registered [ 0.369156] acpiphp: Slot [14] registered [ 0.370120] acpiphp: Slot [15] registered [ 0.372136] acpiphp: Slot [16] registered [ 0.374145] acpiphp: Slot [17] registered [ 0.376152] acpiphp: Slot [18] registered [ 0.377134] acpiphp: Slot [19] registered [ 0.379185] acpiphp: Slot [20] registered [ 0.381119] acpiphp: Slot [21] registered [ 0.382121] acpiphp: Slot [22] registered [ 0.384147] acpiphp: Slot [23] registered [ 0.386135] acpiphp: Slot [24] registered [ 0.387103] acpiphp: Slot [25] registered [ 0.389159] acpiphp: Slot [26] registered [ 0.390143] acpiphp: Slot [27] registered [ 0.392148] acpiphp: Slot [28] registered [ 0.393122] acpiphp: Slot [29] registered [ 0.394153] acpiphp: Slot [30] registered [ 0.396136] acpiphp: Slot [31] registered [ 0.398088] PCI host bridge to bus 0000:00 [ 0.399028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.402035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.404034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.406035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.409033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.413069] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.415257] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.418646] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.421384] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.428018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.432375] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.434022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.435021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.437027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.439724] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.442844] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.445112] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.447816] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.450820] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.462019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.466018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.472860] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.482023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.491023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.512024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.521775] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.527021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.533024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.549037] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.561084] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.567016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.572018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.587022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.596779] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.603974] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.609017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.630022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.639848] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.646014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.651019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.666018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.676277] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.683024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.690023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.707016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.717515] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.721494] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.729330] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.733433] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.736250] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.746073] iommu: Default domain type: Passthrough [ 0.747828] SCSI subsystem initialized [ 0.749193] ACPI: bus type USB registered [ 0.751140] usbcore: registered new interface driver usbfs [ 0.753545] usbcore: registered new interface driver hub [ 0.756105] usbcore: registered new device driver usb [ 0.757213] pps_core: LinuxPPS API ver. 1 registered [ 0.760016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.763074] PTP clock support registered [ 0.765186] EDAC MC: Ver: 3.0.0 [ 0.768198] PCI: Using ACPI for IRQ routing [ 0.771245] NetLabel: Initializing [ 0.772018] NetLabel: domain hash size = 128 [ 0.774022] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.778139] NetLabel: unlabeled traffic allowed by default [ 0.781115] vgaarb: loaded [ 0.782304] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.784025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.791379] clocksource: Switched to clocksource kvm-clock [ 0.900096] VFS: Disk quotas dquot_6.6.0 [ 0.901915] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.904358] *** VALIDATE ramfs *** [ 0.905740] *** VALIDATE hugetlbfs *** [ 0.907746] pnp: PnP ACPI init [ 0.910544] pnp: PnP ACPI: found 6 devices [ 0.929422] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.933439] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.935957] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.938172] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.940532] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.943172] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.946485] NET: Registered protocol family 2 [ 0.949066] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.954137] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.958177] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.964248] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.968452] TCP: Hash tables configured (established 65536 bind 65536) [ 0.971858] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.975470] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.978736] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.982062] NET: Registered protocol family 1 [ 0.985067] RPC: Registered named UNIX socket transport module. [ 0.987683] RPC: Registered udp transport module. [ 0.989601] RPC: Registered tcp transport module. [ 0.991532] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.994342] NET: Registered protocol family 44 [ 0.995955] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.998333] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.000366] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.002125] PCI: CLS 0 bytes, default 64 [ 1.004261] Unpacking initramfs... [ 2.396816] debug: unmapping init [mem 0xffff93d8fcc54000-0xffff93d8fffbffff] [ 2.401046] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.403245] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.406706] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 2.908468] Initialise system trusted keyrings [ 2.910235] Key type blacklist registered [ 2.912465] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.921966] zbud: loaded [ 2.925236] *** VALIDATE nfs *** [ 2.926676] *** VALIDATE nfs4 *** [ 2.928562] pstore: using deflate compression [ 2.932354] Platform Keyring initialized [ 3.075107] NET: Registered protocol family 38 [ 3.077236] Key type asymmetric registered [ 3.078961] Asymmetric key parser 'x509' registered [ 3.081137] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.084793] io scheduler mq-deadline registered [ 3.086817] io scheduler kyber registered [ 3.088695] io scheduler bfq registered [ 3.091599] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.095108] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.098022] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.100897] ACPI: Power Button [PWRF] [ 3.107041] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.114778] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.130233] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.139311] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.155753] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.184413] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.213609] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.220705] Non-volatile memory driver v1.3 [ 3.222223] Linux agpgart interface v0.103 [ 3.260580] virtio_blk virtio1: [vda] 67992 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.263301] vda: detected capacity change from 0 to 34811904 [ 3.276634] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.279815] vdb: detected capacity change from 0 to 1073741824 [ 3.295753] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.299789] vdc: detected capacity change from 0 to 2621440000 [ 3.314706] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.317956] vdd: detected capacity change from 0 to 2621440000 [ 3.334397] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.336862] vde: detected capacity change from 0 to 4294967296 [ 3.353916] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.357536] vdf: detected capacity change from 0 to 4294967296 [ 3.365455] libphy: Fixed MDIO Bus: probed [ 3.370342] usbcore: registered new interface driver usbserial_generic [ 3.373618] usbserial: USB Serial support registered for generic [ 3.376489] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.381595] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.383952] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.386851] mousedev: PS/2 mouse device common for all mice [ 3.389800] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.393285] rtc_cmos 00:05: RTC can wake from S4 [ 3.396693] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.403233] rtc_cmos 00:05: registered as rtc0 [ 3.405952] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.407260] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.408511] intel_pstate: CPU model not supported [ 3.414980] hid: raw HID events driver (C) Jiri Kosina [ 3.417856] usbcore: registered new interface driver usbhid [ 3.420115] usbhid: USB HID core driver [ 3.422265] drop_monitor: Initializing network drop monitor service [ 3.424859] Initializing XFRM netlink socket [ 3.427177] NET: Registered protocol family 10 [ 3.430578] Segment Routing with IPv6 [ 3.432930] NET: Registered protocol family 17 [ 3.435596] mpls_gso: MPLS GSO support [ 3.441857] RAS: Correctable Errors collector initialized. [ 3.443369] AVX version of gcm_enc/dec engaged. [ 3.444423] AES CTR mode by8 optimization enabled [ 3.533965] sched_clock: Marking stable (3533925564, 0)->(4495255109, -961329545) [ 3.538107] registered taskstats version 1 [ 3.541636] Loading compiled-in X.509 certificates [ 3.544273] zswap: loaded using pool lzo/zbud [ 3.574478] Key type big_key registered [ 3.588594] Key type encrypted registered [ 3.592707] ima: No TPM chip found, activating TPM-bypass! [ 3.595634] ima: Allocated hash algorithm: sha1 [ 3.597674] ima: No architecture policies found [ 3.600067] evm: Initialising EVM extended attributes: [ 3.602339] evm: security.selinux [ 3.603810] evm: security.ima [ 3.605136] evm: security.capability [ 3.606726] evm: HMAC attrs: 0x1 [ 3.609357] rtc_cmos 00:05: setting system clock to 2026-04-14 20:47:42 UTC (1776199662) [ 3.615307] debug: unmapping init [mem 0xffffffff88603000-0xffffffff887fffff] [ 3.618508] debug: unmapping init [mem 0xffffffff87382000-0xffffffff87658fff] [ 3.623064] Write protecting the kernel read-only data: 28672k [ 3.626373] debug: unmapping init [mem 0xffffffff85a03000-0xffffffff85bfffff] [ 3.628889] debug: unmapping init [mem 0xffffffff86314000-0xffffffff863fffff] [ 3.672972] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.682424] systemd[1]: Detected virtualization kvm. [ 3.684919] systemd[1]: Detected architecture x86-64. [ 3.687327] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.717941] systemd[1]: No hostname configured. [ 3.720309] systemd[1]: Set hostname to . [ 3.722977] random: systemd: uninitialized urandom read (16 bytes read) [ 3.726281] systemd[1]: Initializing machine ID from random generator. [ 3.780184] random: ln: uninitialized urandom read (6 bytes read) [ 3.863904] random: systemd: uninitialized urandom read (16 bytes read) [ 3.866299] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.870498] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.875672] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.516200] device-mapper: uevent: version 1.0.3 [ 4.518673] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.283939] virtio_net virtio0 ens2: renamed from eth0 [ 5.296838] random: fast init done [ 5.366368] scsi host0: ata_piix [ 5.372255] scsi host1: ata_piix [ 5.373877] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.376861] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.111885] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.111639] random: crng init done [ 10.113242] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.568147] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.695690] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.965631] SELinux: Disabled at runtime. [ 12.032101] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.042173] systemd[1]: Detected virtualization kvm. [ 12.044334] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.518365] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.521603] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.527551] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.533296] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.535735] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.541804] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.547359] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.711640] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.985072] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.248674] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.271879] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.390076] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.408454] EDAC sbridge: Ver: 1.1.2 [ 15.040347] Key type dns_resolver registered [ 15.343659] NFS: Registering the id_resolver key type [ 15.345941] Key type id_resolver registered [ 15.347680] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg318-server login: [ 37.674558] libcfs: loading out-of-tree module taints kernel. [ 37.696196] alg: No test for adler32 (adler32-zlib) [ 38.448406] Key type ._llcrypt registered [ 38.450598] Key type .llcrypt registered [ 38.506575] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 54.492252] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 55.957600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 56.665891] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 57.446584] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 57.455893] LNet: Accept secure, port 988 [ 59.271182] Key type lgssc registered [ 60.657834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.458006] hrtimer: interrupt took 4010877 ns [ 76.479456] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 78.607869] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 90.722032] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.493912] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.223178] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 120.767272] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 133.323279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.391630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.542359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 134.563790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 134.677790] Lustre: lustre-MDT0000: new disk, initializing [ 134.768264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.793288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 139.695155] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 153.528846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.595713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.680689] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 153.705378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 153.715260] Lustre: Skipped 1 previous similar message [ 153.815644] Lustre: lustre-MDT0001: new disk, initializing [ 153.900799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.960729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 153.974043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 158.186768] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 169.769834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 169.844487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 169.854110] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 170.253113] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 170.284201] Lustre: lustre-OST0000: new disk, initializing [ 170.290573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 170.348639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.655244] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 175.615846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 175.625510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 187.960108] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.051323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 188.065967] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 188.183866] Lustre: lustre-OST0001: new disk, initializing [ 188.187479] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 188.258831] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.975338] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 193.571975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 193.581020] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 203.479527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 215.521070] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 223.819181] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing check_logdir /tmp/testlogs/ [ 228.792808] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing yml_node [ 232.989017] Lustre: DEBUG MARKER: Client: 2.15.8.2 [ 235.188973] Lustre: DEBUG MARKER: MDS: 2.15.8.2 [ 237.655425] Lustre: DEBUG MARKER: OSS: 2.15.8.2 [ 239.631343] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 16:51:35 EDT 2026 [ 250.034475] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 252.392223] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 270.312870] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.316299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 270.331863] Lustre: Skipped 2 previous similar messages [ 270.347287] Lustre: Skipped 3 previous similar messages [ 274.867408] Lustre: server umount lustre-MDT0000 complete [ 275.425828] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 275.441897] LustreError: Skipped 3 previous similar messages [ 278.128147] LustreError: 8248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199937 with bad export cookie 13750400074830545627 [ 278.133519] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 280.544187] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.545125] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.557314] Lustre: Skipped 1 previous similar message [ 280.557964] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 280.575848] LustreError: Skipped 1 previous similar message [ 280.594618] Lustre: Skipped 1 previous similar message [ 283.551962] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 283.552473] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 283.569559] LustreError: Skipped 1 previous similar message [ 283.584083] Lustre: Skipped 1 previous similar message [ 284.410392] Lustre: server umount lustre-MDT0001 complete [ 288.363974] Lustre: server umount lustre-OST0000 complete [ 291.818581] Lustre: server umount lustre-OST0001 complete [ 297.773207] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 305.904114] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 314.354300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 320.834726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 326.854582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 331.959047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 338.290907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 338.364187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.580658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 338.639395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 338.732753] Lustre: lustre-MDT0000: new disk, initializing [ 338.800467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 338.823825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 343.075277] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 353.421680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 353.504235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 353.576405] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 353.610668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 353.614133] Lustre: Skipped 1 previous similar message [ 353.686787] Lustre: lustre-MDT0001: new disk, initializing [ 353.780484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 353.810844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 353.828284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 357.883580] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 367.518558] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 373.616427] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 380.189596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 380.255798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 380.260557] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 380.496916] Lustre: lustre-OST0000: new disk, initializing [ 380.502212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 380.572518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.445887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 382.460808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 385.064301] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 394.654682] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 398.321631] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 398.527994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 403.717536] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 403.951691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 408.032081] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 408.033068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.050869] LustreError: Skipped 1 previous similar message [ 408.072361] Lustre: Skipped 1 previous similar message [ 408.077966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 413.158407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 413.164469] Lustre: Skipped 2 previous similar messages [ 420.319611] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 420.691559] Lustre: server umount lustre-OST0000 complete [ 425.440094] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 425.444243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.469162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 425.475244] Lustre: Skipped 2 previous similar messages [ 431.357686] Lustre: server umount lustre-MDT0000 complete [ 433.632464] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 433.654879] LustreError: Skipped 1 previous similar message [ 435.097725] LustreError: 14341:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200093 with bad export cookie 13750400074830547083 [ 435.123747] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 435.134287] LustreError: 14341:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 441.300350] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 16:54:57 (1776200097) [ 449.358378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.828182] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 449.908431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.547224] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 467.051544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.254414] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 483.326587] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 491.098818] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 499.430313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 503.701510] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 511.699095] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 524.007864] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 533.984693] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 534.001212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.011376] Lustre: Skipped 1 previous similar message [ 534.023160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 534.033165] Lustre: Skipped 2 previous similar messages [ 536.367567] Lustre: server umount lustre-MDT0000 complete [ 536.382955] Lustre: Skipped 1 previous similar message [ 539.616257] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.656914] LustreError: Skipped 5 previous similar messages [ 540.765215] LustreError: 18737:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200199 with bad export cookie 13750400074830547937 [ 540.766518] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.775470] LustreError: 18737:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 552.415150] Lustre: 21885:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200205/real 1776200205] req@0000000088b4288e x1862480393491776/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200211 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 573.850577] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 582.569815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.135686] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.241492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.254137] Lustre: Skipped 2 previous similar messages [ 588.203769] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 596.957794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.860707] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 610.441393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 615.700210] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 624.451554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 624.507724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 624.515026] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 624.646542] Lustre: lustre-OST0001: new disk, initializing [ 624.653806] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 626.249866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 626.265355] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 628.892728] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 638.322303] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 641.757265] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 657.393192] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.396355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 657.414047] Lustre: Skipped 4 previous similar messages [ 657.433029] Lustre: Skipped 6 previous similar messages [ 661.935143] Lustre: server umount lustre-MDT0000 complete [ 661.937508] Lustre: Skipped 2 previous similar messages [ 662.505099] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 662.515676] LustreError: Skipped 2 previous similar messages [ 665.763809] LustreError: 26318:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200324 with bad export cookie 13750400074830548784 [ 665.766042] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 665.774801] LustreError: 26318:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 674.719407] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200326/real 1776200326] req@000000001b82d29d x1862480393522624/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200333 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 674.738658] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 674.749893] Lustre: Skipped 1 previous similar message [ 676.831217] Lustre: 27085:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200329/real 1776200329] req@000000006e0bc0ba x1862480393522752/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200335 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 678.880473] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200330/real 1776200330] req@0000000024e687ad x1862480393522944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200337 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 678.920080] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 697.038904] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 706.088952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.704656] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 720.859494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.398330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 721.414158] Lustre: Skipped 4 previous similar messages [ 727.554687] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 740.273602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 747.045492] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 759.099151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 762.242814] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 769.088618] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 772.952158] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 772.958878] Lustre: Skipped 1 previous similar message [ 784.865540] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.871608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 784.886272] Lustre: Skipped 2 previous similar messages [ 784.904839] Lustre: Skipped 3 previous similar messages [ 790.119708] Lustre: server umount lustre-MDT0000 complete [ 790.128108] Lustre: Skipped 3 previous similar messages [ 793.428629] LustreError: 28500:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200452 with bad export cookie 13750400074830550030 [ 793.433952] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 793.435511] LustreError: 28500:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 801.183617] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200453/real 1776200453] req@00000000b1e08d1e x1862480393549312/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200460 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 801.228487] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 818.401721] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 17:01:14 (1776200474) [ 848.631687] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 861.086568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.690813] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 861.736815] LustreError: Skipped 7 previous similar messages [ 866.381158] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 875.188921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.694880] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 888.676505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 893.766555] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 902.227536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 907.310509] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 915.043783] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 919.035617] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 932.320877] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 932.326666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.350688] Lustre: Skipped 2 previous similar messages [ 932.366028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 932.369065] Lustre: Skipped 4 previous similar messages [ 935.936536] Lustre: server umount lustre-MDT0000 complete [ 935.943250] Lustre: Skipped 3 previous similar messages [ 939.796737] LustreError: 34361:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200598 with bad export cookie 13750400074830551094 [ 939.802401] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 939.804726] LustreError: 34361:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 949.219186] Lustre: 37937:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200602/real 1776200602] req@0000000091f86aab x1862480393575872/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200608 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 949.264724] Lustre: 37937:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 961.164472] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 17:03:37 (1776200617) [ 975.353919] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 984.815528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.238357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.255231] Lustre: Skipped 6 previous similar messages [ 989.846583] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1000.165397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.476468] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1015.229936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1020.169444] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1029.032193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1033.805626] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1040.821459] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1050.126298] Lustre: Setting parameter general.mdd.*.changelog_mask in log params [ 1050.137422] Lustre: Skipped 1 previous similar message [ 1063.066080] LustreError: 39563:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200721 with bad export cookie 13750400074830552165 [ 1063.070325] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1063.077496] LustreError: 39563:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1066.468948] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1066.493549] Lustre: Skipped 8 previous similar messages [ 1078.771920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.801274] Lustre: Evicted from MGS (at 192.168.203.118@tcp) after server handle changed from 0xbed33c03ebc79c65 to 0xbed33c03ebc7a0cc [ 1080.823517] Lustre: MGC192.168.203.118@tcp: Connection restored to (at 0@lo) [ 1081.247748] BUG: spinlock bad magic on CPU#2, lctl/43498 [ 1081.253152] lock: 0xffff93d841ca6900, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 1081.257194] CPU: 2 PID: 43498 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1081.266779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 1081.272538] Call Trace: [ 1081.276281] ? dump_stack+0xbb/0x10e [ 1081.277614] ? spin_dump.cold.3+0x32/0x65 [ 1081.279049] ? do_raw_spin_lock+0xce/0x190 [ 1081.282177] ? _raw_spin_lock+0x21/0x30 [ 1081.285257] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 1081.289335] ? __seccomp_filter+0x4b/0x680 [ 1081.290791] ? full_proxy_write+0x5e/0xa0 [ 1081.296599] ? __vfs_write+0x1c/0x60 [ 1081.300868] ? vfs_write+0xd8/0x2b0 [ 1081.304649] ? ksys_write+0x66/0x120 [ 1081.307207] ? __x64_sys_write+0x1e/0x30 [ 1081.309671] ? do_syscall_64+0xc1/0x440 [ 1081.311071] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1086.136862] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1094.633399] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1095.116888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1101.305305] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1101.330191] Lustre: Skipped 1 previous similar message [ 1101.469945] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1110.428229] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1112.700052] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1119.213698] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1121.332545] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1131.492087] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1135.014455] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.029747] LustreError: Skipped 41 previous similar messages [ 1136.720641] LustreError: 39563:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200795 with bad export cookie 13750400074830553292 [ 1136.721584] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1136.728915] LustreError: 39563:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1146.335165] Lustre: 45716:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200799/real 1776200799] req@00000000e3ad7dbe x1862480393621824/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200805 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1146.356829] Lustre: 45716:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1159.740279] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 17:06:56 (1776200816) [ 1180.927478] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1193.976791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.392183] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1210.753517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.068427] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1225.105126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1231.048765] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1239.085777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1243.795346] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1251.570484] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1275.878405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1275.885519] Lustre: Skipped 23 previous similar messages [ 1281.619927] Lustre: server umount lustre-MDT0000 complete [ 1281.622208] Lustre: Skipped 9 previous similar messages [ 1286.296026] LustreError: 50353:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200945 with bad export cookie 13750400074830554104 [ 1286.298606] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1286.310431] LustreError: 50353:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1298.335250] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200950/real 1776200950] req@000000002d55dd3f x1862480393651328/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200957 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 1298.356981] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1313.019823] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 17:09:29 (1776200969) [ 1315.839847] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1317.995445] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 17:09:34 (1776200974) [ 1329.200733] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 1338.625046] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1347.471615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1353.550411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1360.189115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1360.272557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.562455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1360.566637] Lustre: Skipped 2 previous similar messages [ 1360.608342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1360.673146] Lustre: lustre-MDT0000: new disk, initializing [ 1360.766734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1365.130612] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1373.875136] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1380.238853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1380.295269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1380.299695] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1380.477439] Lustre: lustre-OST0000: new disk, initializing [ 1380.481899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1380.492383] Lustre: Skipped 1 previous similar message [ 1383.685955] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1391.219982] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1393.252753] Lustre: DEBUG MARKER: create test files [ 1417.698968] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1417.701527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.724255] Lustre: Skipped 10 previous similar messages [ 1428.449167] Lustre: 55662:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201081/real 1776201081] req@0000000060b791f6 x1862480393660288/t0(0) o251->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776201087 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1428.478807] Lustre: 55662:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1439.950708] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 1442.604179] Key type lgssc unregistered [ 1442.863664] LNet: 56265:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1442.875044] LNet: Removed LNI 192.168.203.118@tcp [ 1443.645947] Key type .llcrypt unregistered [ 1443.648599] Key type ._llcrypt unregistered [ 1468.128722] alg: No test for adler32 (adler32-zlib) [ 1468.882663] Key type ._llcrypt registered [ 1468.887776] Key type .llcrypt registered [ 1468.997988] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1470.014676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1470.273440] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 1470.488485] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 1470.493983] LNet: Accept secure, port 988 [ 1472.127265] Key type lgssc registered [ 1472.949317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1480.109248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1481.536894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1486.020308] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1493.159991] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1500.547729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1500.790631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.595257] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1512.050239] Lustre: lustre-OST0000: deleting orphan objects from 0x0:102 to 0x0:129 [ 1512.138723] Lustre: 59682:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1512.153716] Lustre: 59682:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1512.165654] Lustre: 59682:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1512.177251] Lustre: 59682:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1512.183396] Lustre: 59682:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1512.190535] Lustre: 59682:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1513.209488] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1517.993972] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1523.201550] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1531.534788] Lustre: server umount lustre-OST0000 complete [ 1540.575185] Lustre: 60120:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201193/real 1776201193] req@000000007a3dd15e x1862481875048384/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776201199 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1546.210500] Lustre: 60120:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201199/real 1776201199] req@00000000ef52f840 x1862481875048768/t0(0) o251->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776201205 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1546.576659] Lustre: server umount lustre-MDT0000 complete [ 1558.717293] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 1561.626124] Key type lgssc unregistered [ 1561.920650] LNet: 60724:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1561.943914] LNet: Removed LNI 192.168.203.118@tcp [ 1562.776671] Key type .llcrypt unregistered [ 1562.782102] Key type ._llcrypt unregistered [ 1591.095736] alg: No test for adler32 (adler32-zlib) [ 1591.857759] Key type ._llcrypt registered [ 1591.861789] Key type .llcrypt registered [ 1591.971796] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1593.077881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1593.492104] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 1593.742472] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 1593.748981] LNet: Accept secure, port 988 [ 1595.463186] Key type lgssc registered [ 1596.478722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1603.414933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.804341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.166941] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1616.298381] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1622.092734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1622.379515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.199814] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1633.208826] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1635.649082] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 1635.721905] Lustre: 63659:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1635.731156] Lustre: 63659:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1635.739296] Lustre: 63659:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1635.745622] Lustre: 63659:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1635.754299] Lustre: 63659:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1635.763726] Lustre: 63659:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1644.596917] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1651.171268] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1651.193713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.205988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.290458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.848158] Lustre: server umount lustre-OST0000 complete [ 1668.064447] Lustre: 64637:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201320/real 1776201320] req@00000000bb83e234 x1862482004023680/t0(0) o251->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776201326 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1668.298799] Lustre: server umount lustre-MDT0000 complete [ 1681.591617] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 1684.059850] Key type lgssc unregistered [ 1684.332439] LNet: 65241:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.354156] LNet: Removed LNI 192.168.203.118@tcp [ 1685.049207] Key type .llcrypt unregistered [ 1685.054667] Key type ._llcrypt unregistered [ 1702.912737] alg: No test for adler32 (adler32-zlib) [ 1703.672450] Key type ._llcrypt registered [ 1703.678066] Key type .llcrypt registered [ 1703.771454] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 1715.604138] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1716.612495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1717.284327] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 1717.557735] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 1717.561868] LNet: Accept secure, port 988 [ 1719.207742] Key type lgssc registered [ 1720.272954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1728.847257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1734.716747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1739.661704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1745.856355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1752.808110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1752.914177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.158713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1754.197826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1754.337833] Lustre: lustre-MDT0000: new disk, initializing [ 1754.504434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.524351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1759.254510] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1768.510041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1768.576130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.682913] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1768.725793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1768.735684] Lustre: Skipped 1 previous similar message [ 1768.873639] Lustre: lustre-MDT0001: new disk, initializing [ 1768.945087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1768.982804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1768.996219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1773.307606] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1782.137860] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1787.446564] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1794.296359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1794.384562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1794.390775] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1794.604401] Lustre: lustre-OST0000: new disk, initializing [ 1794.608159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1794.682654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.179433] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1802.304432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1802.311502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1808.339692] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1813.281903] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1813.511700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1817.869299] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1818.063449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1823.203705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.210347] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1823.220067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1823.241924] Lustre: Skipped 1 previous similar message [ 1825.978884] Lustre: server umount lustre-OST0000 complete [ 1830.368081] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1830.374162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.383696] Lustre: Skipped 1 previous similar message [ 1830.386696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.440683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.441613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.768816] Lustre: server umount lustre-MDT0000 complete [ 1838.561112] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1838.588517] LustreError: Skipped 1 previous similar message [ 1839.413663] LustreError: 68515:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201498 with bad export cookie 14897400859299794577 [ 1839.414261] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1839.420158] LustreError: 68515:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1839.743823] Lustre: server umount lustre-MDT0001 complete [ 1847.480958] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 17:18:23 (1776201503) [ 1857.676566] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1871.140967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1899.626089] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 1908.786896] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1917.898560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1924.410270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1930.000200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1936.134947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1942.244950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1942.329232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.587172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1942.621099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1942.675244] Lustre: lustre-MDT0000: new disk, initializing [ 1942.743247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.757976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1947.070575] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1957.467990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1957.518623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.604026] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1957.642228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1957.646151] Lustre: Skipped 1 previous similar message [ 1957.734549] Lustre: lustre-MDT0001: new disk, initializing [ 1957.806616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1957.839409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1957.854410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1961.819707] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1970.691167] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1976.136209] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1982.150402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1982.211374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1982.219884] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1982.443413] Lustre: lustre-OST0000: new disk, initializing [ 1982.449395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1982.521516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.639241] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 1988.649581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1988.661573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1995.913393] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2000.272179] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2000.603742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2004.677798] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2004.864885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2009.059354] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2009.066455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.079653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2009.088854] Lustre: Skipped 1 previous similar message [ 2012.625789] Lustre: server umount lustre-OST0000 complete [ 2019.296154] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2019.307068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.317736] Lustre: Skipped 1 previous similar message [ 2019.322164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.330760] Lustre: Skipped 1 previous similar message [ 2022.045793] Lustre: server umount lustre-MDT0000 complete [ 2024.933415] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.949358] LustreError: Skipped 1 previous similar message [ 2025.252071] LustreError: 76809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201684 with bad export cookie 14897400859299795431 [ 2025.254526] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.275332] LustreError: 76809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2034.525478] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 17:21:30 (1776201690) [ 2041.038089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.524352] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.612996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.895732] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2046.955206] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.064996] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.804095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.791895] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2067.063405] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2073.172610] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2080.761423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2081.085085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.093851] Lustre: Skipped 1 previous similar message [ 2085.780192] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2094.125838] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2097.914090] Lustre: 84108:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting 7eb964e1-5604-4295-81a9-ca799c80cfbf at adminstrative request [ 2102.781125] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2102.797598] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2108.897190] Lustre: MGS: Client 7eb964e1-5604-4295-81a9-ca799c80cfbf (at 0@lo) reconnecting [ 2108.915355] Lustre: Evicted from MGS (at 192.168.203.118@tcp) after server handle changed from 0xcebe3326d729d53d to 0xcebe3326d729d7dd [ 2108.934368] Lustre: MGC192.168.203.118@tcp: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 2135.348672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2135.453520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2135.465112] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2135.603991] Lustre: lustre-OST0001: new disk, initializing [ 2135.613721] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2140.476663] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2142.294392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2142.309528] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2150.970693] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2158.048649] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2158.054845] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.060678] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2158.074425] Lustre: Skipped 2 previous similar messages [ 2158.084465] Lustre: Skipped 1 previous similar message [ 2168.294364] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2168.351073] Lustre: server umount lustre-OST0001 complete [ 2168.359839] Lustre: Skipped 1 previous similar message [ 2173.409207] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2173.423203] LustreError: Skipped 1 previous similar message [ 2178.532223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2178.540330] Lustre: Skipped 8 previous similar messages [ 2186.207620] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2186.383741] Lustre: server umount lustre-OST0000 complete [ 2193.376338] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2193.396848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.408864] Lustre: Skipped 2 previous similar messages [ 2199.009392] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.026987] LustreError: Skipped 5 previous similar messages [ 2199.103989] LustreError: 85520:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201857 with bad export cookie 14897400859299796957 [ 2199.111599] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.112370] LustreError: 85520:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2207.308539] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 17:24:23 (1776201863) [ 2208.818385] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2210.566793] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 17:24:27 (1776201867) [ 2212.278591] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2214.143668] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 17:24:30 (1776201870) [ 2215.766617] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2217.943519] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 17:24:34 (1776201874) [ 2219.624442] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2224.886753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2225.871761] loop: module loaded [ 2225.894125] loop0: detected capacity change from 0 to 4294967296 [ 2225.981386] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2227.592567] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2231.325968] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2235.425656] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2242.962541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2246.242523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2249.636940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2252.792446] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2258.700855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2258.766831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2258.773081] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2258.781681] LustreError: 88559:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2258.789392] LustreError: 88559:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2258.800916] LustreError: 88559:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2258.812102] LustreError: 88559:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2275.506822] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 2284.473794] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 2294.675435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2302.160217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2308.569694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2314.862975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2322.039799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2322.095712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.349534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2322.374342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2322.446166] Lustre: lustre-MDT0000: new disk, initializing [ 2322.595130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.599163] Lustre: Skipped 1 previous similar message [ 2322.627203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2327.620294] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2338.218312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2338.300959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.423294] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2338.796180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2343.468230] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2352.083386] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2357.415341] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2363.582358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2363.654856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2363.664702] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2367.882473] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2372.700310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2377.627973] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2382.573437] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2382.853829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2387.507577] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2387.755809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2393.570685] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2393.575579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.580569] LustreError: Skipped 1 previous similar message [ 2393.582891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2393.582898] Lustre: Skipped 4 previous similar messages [ 2393.635780] Lustre: Skipped 2 previous similar messages [ 2395.890309] Lustre: server umount lustre-OST0000 complete [ 2395.894145] Lustre: Skipped 2 previous similar messages [ 2414.047360] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2415.073641] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.106483] LustreError: Skipped 1 previous similar message [ 2418.726849] LustreError: 91427:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202077 with bad export cookie 14897400859299797538 [ 2418.737509] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2418.745787] LustreError: 91427:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2430.456831] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 17:28:06 (1776202086) [ 2437.406491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.678842] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2452.860897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.217162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2453.220205] Lustre: Skipped 3 previous similar messages [ 2458.762814] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2467.526097] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2474.186829] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2482.884723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2490.057235] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2498.853650] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2505.894853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2505.977752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2505.985857] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2506.080603] Lustre: lustre-OST0001: new disk, initializing [ 2506.082216] Lustre: Skipped 2 previous similar messages [ 2506.084258] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2506.086759] Lustre: Skipped 3 previous similar messages [ 2510.567835] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2516.056636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2516.070449] Lustre: Skipped 2 previous similar messages [ 2516.096120] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2519.169893] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2536.213268] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2558.686490] Lustre: Failing over lustre-MDT0000 [ 2560.375297] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 2560.383604] Lustre: Skipped 9 previous similar messages [ 2560.996124] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2561.001187] LustreError: Skipped 1 previous similar message [ 2561.010082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2561.031368] Lustre: Skipped 2 previous similar messages [ 2561.413996] Lustre: server umount lustre-MDT0000 complete [ 2561.424031] Lustre: Skipped 2 previous similar messages [ 2562.534575] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.557164] LustreError: Skipped 6 previous similar messages [ 2569.567501] Lustre: 66741:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202221/real 1776202221] req@00000000f3aac0db x1862482134169664/t0(0) o400->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202228 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2569.602961] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2588.431823] LDISKFS-fs (dm-0): recovery complete [ 2588.435877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.470785] Lustre: Evicted from MGS (at 192.168.203.118@tcp) after server handle changed from 0xcebe3326d729dd78 to 0xcebe3326d72a6b8c [ 2602.493614] Lustre: MGC192.168.203.118@tcp: Connection restored to (at 0@lo) [ 2602.746352] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2603.850668] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2603.861776] Lustre: 95982:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2606.892097] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2608.098673] Lustre: 95986:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2608.109098] Lustre: 95986:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 1 previous similar message [ 2608.131549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2608.208890] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2608.535089] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2608.865260] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2608.878122] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2609.199142] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2609.203929] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2609.209356] Lustre: 101174:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2609.219607] Lustre: 101174:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 7 previous similar messages [ 2610.172098] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2610.180486] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 2610.506327] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2610.516086] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 2611.484153] Lustre: 101174:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2611.497135] Lustre: 101174:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 6 previous similar messages [ 2612.459192] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2612.471381] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2612.791290] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2612.801479] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2615.639878] Lustre: 101174:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2615.656829] Lustre: 101174:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages [ 2616.617397] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2616.631333] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2616.951178] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2616.959667] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2623.892697] Lustre: 101174:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2623.914546] Lustre: 101174:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 25 previous similar messages [ 2624.892323] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2624.903203] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 2625.228247] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2625.235534] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 2640.073576] Lustre: 101174:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2640.085724] Lustre: 101174:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 50 previous similar messages [ 2641.033973] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2641.049121] LustreError: 101174:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2641.367121] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2641.372650] LustreError: 101174:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2663.136212] Lustre: 101174:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2663.153046] Lustre: 101174:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2663.165590] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2663.170272] LustreError: 101174:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 2663.189890] Lustre: 101174:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@0000000032391d3f x1862482120437568/t0(8589934770) o35->b45cfa16-e965-430c-b0ed-1bc3434ed163@192.168.203.18@tcp:652/0 lens 392/0 e 0 to 0 dl 1776202327 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 2663.248659] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 2663.252546] Lustre: Skipped 2 previous similar messages [ 2663.259071] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2663.315774] Lustre: lustre-OST0000: deleting orphan objects from 0x0:78 to 0x0:577 [ 2663.320524] Lustre: lustre-OST0001: deleting orphan objects from 0x0:45 to 0x0:545 [ 2679.264436] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2679.272311] LustreError: Skipped 1 previous similar message [ 2689.514069] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.538520] Lustre: Skipped 5 previous similar messages [ 2689.543473] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2689.548036] Lustre: Skipped 3 previous similar messages [ 2692.773259] Lustre: server umount lustre-OST0001 complete [ 2692.779678] Lustre: Skipped 1 previous similar message [ 2705.377767] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.391111] LustreError: Skipped 65 previous similar messages [ 2706.169166] LustreError: 95968:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202365 with bad export cookie 14897400859299834764 [ 2706.171837] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2706.180439] LustreError: 95968:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2714.054915] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 17:32:50 (1776202370) [ 2721.095790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2721.102718] Lustre: *** cfs_fail_loc=197, val=0*** [ 2738.655671] LustreError: 103199:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.118@tcp: failed processing log, type 1: rc = -5 [ 2749.919700] LustreError: 103199:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.118@tcp: failed processing log, type 4: rc = -110 [ 2755.062971] Lustre: *** cfs_fail_loc=197, val=0*** [ 2755.068400] Lustre: Skipped 65 previous similar messages [ 2778.719542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.728489] Lustre: Skipped 3 previous similar messages [ 2782.146971] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2789.175903] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2798.471382] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 17:34:14 (1776202454) [ 2820.099613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.966632] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2834.525591] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2838.423853] LustreError: 104858:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2838.432692] LustreError: 104858:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2838.439825] LustreError: 104858:0:(lod_dev.c:437:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 2844.639189] Lustre: 105548:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202497/real 1776202497] req@00000000bab1f348 x1862482134204992/t0(0) o251->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202503 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2858.022265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2873.902480] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 2882.708899] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 2891.977642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2898.301648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2903.069073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2908.879193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2915.818977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2915.908479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2916.165974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2916.205557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2916.301958] Lustre: lustre-MDT0000: new disk, initializing [ 2916.403056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2920.463121] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2930.928675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2930.990754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.439417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2935.901567] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2945.685927] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2951.478899] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2957.008502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2957.069209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2957.076376] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2960.684659] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 2969.529786] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2973.640734] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2973.860587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2977.891756] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2978.069536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2982.888604] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2982.889090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.897651] Lustre: Skipped 3 previous similar messages [ 2982.909136] LustreError: Skipped 3 previous similar messages [ 2982.910700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2982.929317] Lustre: Skipped 6 previous similar messages [ 2994.143251] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2994.348808] Lustre: server umount lustre-OST0000 complete [ 2994.353086] Lustre: Skipped 4 previous similar messages [ 3006.436900] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3006.455650] LustreError: Skipped 5 previous similar messages [ 3007.974575] LustreError: 109176:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202666 with bad export cookie 14897400859299845299 [ 3007.986320] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.986847] LustreError: 109176:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3017.041580] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 17:37:53 (1776202673) [ 3027.225994] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3029.671817] Key type lgssc unregistered [ 3029.973833] LNet: 113863:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3029.991304] LNet: Removed LNI 192.168.203.118@tcp [ 3030.745360] Key type .llcrypt unregistered [ 3030.748714] Key type ._llcrypt unregistered [ 3047.794396] alg: No test for adler32 (adler32-zlib) [ 3048.551552] Key type ._llcrypt registered [ 3048.558791] Key type .llcrypt registered [ 3048.715513] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 3061.348282] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3062.321705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3062.692155] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3062.841624] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 3062.843966] LNet: Accept secure, port 988 [ 3064.471122] Key type lgssc registered [ 3065.476132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3074.243110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3081.421947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3088.036708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3095.451901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3102.515703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3109.741213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3109.822565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.001200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3111.062252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3111.201597] Lustre: lustre-MDT0000: new disk, initializing [ 3111.370873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.399887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3117.172197] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3128.967970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3129.038070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3129.048458] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3129.347461] Lustre: lustre-OST0000: new disk, initializing [ 3129.351186] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3129.362943] Lustre: Skipped 1 previous similar message [ 3129.434872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3134.203944] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3145.234049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3145.307193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3145.314290] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3145.452652] Lustre: lustre-OST0001: new disk, initializing [ 3145.456952] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3145.542455] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3150.055862] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3162.599241] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.601702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3162.615820] Lustre: Skipped 1 previous similar message [ 3162.633126] Lustre: Skipped 1 previous similar message [ 3164.923764] Lustre: server umount lustre-MDT0000 complete [ 3171.425651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3174.879574] Lustre: 115362:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202826/real 1776202826] req@000000005b9c741a x1862483544387328/t0(0) o400->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202833 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3174.915409] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3175.950085] Lustre: server umount lustre-OST0000 complete [ 3180.096285] Lustre: server umount lustre-OST0001 complete [ 3189.480226] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 17:40:45 (1776202845) [ 3197.840639] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3205.898788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3206.987067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3212.349830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3213.319650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3221.479566] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 17:41:17 (1776202877) [ 3236.783124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3239.258589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3241.535635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3243.853502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3244.898104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3246.841237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3263.480311] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 3271.767641] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3281.935175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3288.277607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3294.307601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3300.491555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3306.846708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3306.932786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.117713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3307.145929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3307.235699] Lustre: lustre-MDT0000: new disk, initializing [ 3307.323757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.350681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3311.587982] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3321.909038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3321.990858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.101371] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3322.135652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3322.139232] Lustre: Skipped 1 previous similar message [ 3322.216233] Lustre: lustre-MDT0001: new disk, initializing [ 3322.301566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3322.358951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3322.377517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3326.061842] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3334.788798] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3340.799697] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3346.388626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3346.453866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3346.465137] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3346.655267] Lustre: lustre-OST0000: new disk, initializing [ 3346.660635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3346.723578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.151961] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3354.132124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3354.136588] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3359.957689] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3366.203753] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3366.438271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3371.332697] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3371.627661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3374.563553] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3374.567691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3374.583026] LustreError: Skipped 1 previous similar message [ 3374.586676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3374.595078] Lustre: Skipped 1 previous similar message [ 3374.621026] Lustre: Skipped 1 previous similar message [ 3379.681165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3379.690727] Lustre: Skipped 1 previous similar message [ 3384.821493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3384.836468] Lustre: Skipped 1 previous similar message [ 3388.383147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3388.523342] Lustre: server umount lustre-OST0000 complete [ 3394.020357] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3394.031257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.047142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.283473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.291649] Lustre: Skipped 5 previous similar messages [ 3406.303225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3408.429807] Lustre: server umount lustre-MDT0000 complete [ 3410.411536] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.427764] LustreError: Skipped 1 previous similar message [ 3412.031085] LustreError: 125893:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203070 with bad export cookie 14296893305436609843 [ 3412.037555] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3412.047337] LustreError: 125893:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3421.723099] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 17:44:38 (1776203078) [ 3428.267789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3428.829080] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3428.908759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.118199] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3433.957670] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.072665] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.084824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3445.865778] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3454.589410] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3461.097668] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3467.851705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3468.371455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3468.377727] Lustre: Skipped 1 previous similar message [ 3473.660630] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3482.632745] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3487.388956] Lustre: *** cfs_fail_loc=159, val=0*** [ 3494.757659] Lustre: lustre-MDT0000: Client 67ca108d-6a62-4d24-82ae-d4ba67e7bb67 (at 192.168.203.18@tcp) reconnecting [ 3495.911665] Lustre: *** cfs_fail_loc=159, val=0*** [ 3495.920850] Lustre: Skipped 6 previous similar messages [ 3502.402422] Lustre: lustre-MDT0000: Client 67ca108d-6a62-4d24-82ae-d4ba67e7bb67 (at 192.168.203.18@tcp) reconnecting [ 3507.179263] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3507.187055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.205037] Lustre: Skipped 1 previous similar message [ 3507.210529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3508.835571] Lustre: server umount lustre-OST0000 complete [ 3508.838392] Lustre: Skipped 1 previous similar message [ 3513.315509] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3513.322108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.943776] Lustre: 133506:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203171/real 1776203171] req@00000000d564c4e8 x1862483544426176/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776203177 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3519.461490] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3519.498229] LustreError: Skipped 1 previous similar message [ 3523.116923] LustreError: 130436:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203182 with bad export cookie 14296893305436610704 [ 3523.132837] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3523.137915] LustreError: 130436:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3534.372948] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3537.086830] Key type lgssc unregistered [ 3537.377147] LNet: 134311:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3537.385767] LNet: Removed LNI 192.168.203.118@tcp [ 3538.171849] Key type .llcrypt unregistered [ 3538.172990] Key type ._llcrypt unregistered [ 3560.208901] alg: No test for adler32 (adler32-zlib) [ 3560.965723] Key type ._llcrypt registered [ 3560.971070] Key type .llcrypt registered [ 3561.103279] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3561.814023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3562.349396] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3562.523968] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 3562.528337] LNet: Accept secure, port 988 [ 3564.207114] Key type lgssc registered [ 3565.391386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3572.572605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.949447] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3574.000344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3578.171895] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3579.373433] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3584.486038] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.680283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3587.145789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3592.095777] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3600.362787] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3606.440643] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3614.371701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3614.879269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.452969] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3625.168647] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3628.093990] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3637.391483] Lustre: *** cfs_fail_loc=159, val=0*** [ 3644.755837] Lustre: lustre-MDT0000: Client b5566585-1f47-4961-b87c-d70450d2aab0 (at 192.168.203.18@tcp) reconnecting [ 3646.697840] Lustre: *** cfs_fail_loc=159, val=0*** [ 3653.996307] Lustre: lustre-MDT0001: Client b5566585-1f47-4961-b87c-d70450d2aab0 (at 192.168.203.18@tcp) reconnecting [ 3654.972318] Lustre: *** cfs_fail_loc=159, val=0*** [ 3654.978958] Lustre: Skipped 3 previous similar messages [ 3661.177546] Lustre: lustre-MDT0001: Client b5566585-1f47-4961-b87c-d70450d2aab0 (at 192.168.203.18@tcp) reconnecting [ 3669.198594] Lustre: *** cfs_fail_loc=159, val=0*** [ 3669.208927] Lustre: Skipped 6 previous similar messages [ 3676.478373] Lustre: lustre-MDT0000: Client 3094f252-ae82-4912-a467-4f2935b7f1f3 (at 192.168.203.18@tcp) reconnecting [ 3677.840597] Lustre: *** cfs_fail_loc=159, val=0*** [ 3677.842367] Lustre: Skipped 13 previous similar messages [ 3684.164071] Lustre: lustre-MDT0000: Client 3094f252-ae82-4912-a467-4f2935b7f1f3 (at 192.168.203.18@tcp) reconnecting [ 3697.119966] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3697.127779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3697.141093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3697.160846] Lustre: Skipped 1 previous similar message [ 3697.180397] Lustre: Skipped 1 previous similar message [ 3698.285971] Lustre: server umount lustre-OST0000 complete [ 3704.802132] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3704.815380] LustreError: Skipped 1 previous similar message [ 3704.826293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.851778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3707.363499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3707.369174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3708.585681] Lustre: server umount lustre-MDT0000 complete [ 3712.479919] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3712.495328] LustreError: Skipped 1 previous similar message [ 3713.086629] LustreError: 135756:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203371 with bad export cookie 18377231364293770433 [ 3713.091401] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3713.100471] LustreError: 135756:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3713.540910] Lustre: server umount lustre-MDT0001 complete [ 3725.759384] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3729.057861] Key type lgssc unregistered [ 3729.377220] LNet: 140166:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3729.387521] LNet: Removed LNI 192.168.203.118@tcp [ 3730.355544] Key type .llcrypt unregistered [ 3730.365067] Key type ._llcrypt unregistered [ 3754.879901] alg: No test for adler32 (adler32-zlib) [ 3755.635233] Key type ._llcrypt registered [ 3755.642104] Key type .llcrypt registered [ 3755.788497] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3757.029824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3757.678583] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3757.939080] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 3757.946449] LNet: Accept secure, port 988 [ 3759.655158] Key type lgssc registered [ 3760.774868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3769.003599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3770.433507] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3770.485101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3775.325884] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3775.979634] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3781.095967] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3783.954649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.345799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3788.757203] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3796.557446] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3802.242968] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3809.502582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3809.881919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3815.037497] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 3815.163097] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3820.176534] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:129 [ 3823.417647] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3834.644813] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 3835.524732] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight in log params [ 3839.922800] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 3851.238952] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3851.238976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3851.252451] LustreError: Skipped 1 previous similar message [ 3851.254442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3851.268104] Lustre: Skipped 1 previous similar message [ 3851.295123] Lustre: Skipped 1 previous similar message [ 3852.481416] Lustre: server umount lustre-OST0000 complete [ 3860.966433] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3860.977547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.990545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3862.939055] Lustre: server umount lustre-MDT0000 complete [ 3866.592843] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3866.609448] LustreError: Skipped 1 previous similar message [ 3866.926920] LustreError: 141623:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203525 with bad export cookie 2321202233335536891 [ 3866.945775] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3867.332692] Lustre: server umount lustre-MDT0001 complete [ 3880.872943] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3884.033372] Key type lgssc unregistered [ 3884.423870] LNet: 145707:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3884.437355] LNet: Removed LNI 192.168.203.118@tcp [ 3885.284812] Key type .llcrypt unregistered [ 3885.288836] Key type ._llcrypt unregistered [ 3906.927865] alg: No test for adler32 (adler32-zlib) [ 3907.682724] Key type ._llcrypt registered [ 3907.684827] Key type .llcrypt registered [ 3907.778124] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3908.476725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3908.918160] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3909.101043] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 3909.106474] LNet: Accept secure, port 988 [ 3910.735169] Key type lgssc registered [ 3911.598991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3919.508169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3921.151644] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3921.278331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.497542] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3926.814875] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3931.619045] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3936.361788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3936.777327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3941.181727] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3952.530162] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3958.122925] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3964.954044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3965.304677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3966.315848] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 3969.764253] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 3972.462828] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:161 [ 3977.449967] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3982.533763] Lustre: *** cfs_fail_loc=159, val=0*** [ 3989.839179] Lustre: lustre-MDT0000: Client 34956aae-c6ba-4dba-a5e3-447cec348eb4 (at 192.168.203.18@tcp) reconnecting [ 3996.133743] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3996.134774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.140487] LustreError: Skipped 2 previous similar messages [ 3996.198884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3997.152220] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3997.166096] LustreError: Skipped 4 previous similar messages [ 3997.171995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.193146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4000.207705] Lustre: server umount lustre-OST0000 complete [ 4008.417486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.423893] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4008.429759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4008.446420] Lustre: Skipped 3 previous similar messages [ 4010.850081] Lustre: server umount lustre-MDT0000 complete [ 4013.536623] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4013.544896] LustreError: Skipped 1 previous similar message [ 4015.263244] LustreError: 147154:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203674 with bad export cookie 2743535431668730036 [ 4015.279469] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4015.282534] LustreError: 147154:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4015.639040] Lustre: server umount lustre-MDT0001 complete [ 4027.128779] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 4029.713649] Key type lgssc unregistered [ 4030.047687] LNet: 150934:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4030.069194] LNet: Removed LNI 192.168.203.118@tcp [ 4030.903189] Key type .llcrypt unregistered [ 4030.906715] Key type ._llcrypt unregistered [ 4053.124332] alg: No test for adler32 (adler32-zlib) [ 4053.883070] Key type ._llcrypt registered [ 4053.888810] Key type .llcrypt registered [ 4054.030734] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4055.018704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4055.775165] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 4056.050062] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 4056.054053] LNet: Accept secure, port 988 [ 4057.735124] Key type lgssc registered [ 4059.041531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4066.709699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4068.352772] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.427694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4073.161530] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4073.441653] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.566883] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4081.283942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4081.916971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4086.457872] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4094.995893] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4101.588822] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4108.734552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4109.067323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4111.158948] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 4113.404467] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4118.322967] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 4121.854595] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4141.319488] Lustre: lustre-MDT0001: evicting 6959c889-d820-44f5-9dca-e384692f31fb (at 192.168.203.18@tcp) by administrative request [ 4141.326571] Lustre: Skipped 1 previous similar message [ 4172.774484] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4172.788426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.810817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4174.820192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4174.826468] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4174.830150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4176.264124] Lustre: server umount lustre-OST0000 complete [ 4184.544542] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4184.557992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.576684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4184.580658] Lustre: Skipped 1 previous similar message [ 4186.544069] Lustre: server umount lustre-MDT0000 complete [ 4190.176602] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4190.200349] LustreError: Skipped 1 previous similar message [ 4190.330412] LustreError: 152378:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203849 with bad export cookie 8036260026048375354 [ 4190.336880] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4190.345633] LustreError: 152378:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4190.818733] Lustre: server umount lustre-MDT0001 complete [ 4202.032535] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 4204.761513] Key type lgssc unregistered [ 4205.106792] LNet: 156595:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4205.121124] LNet: Removed LNI 192.168.203.118@tcp [ 4205.888598] Key type .llcrypt unregistered [ 4205.890253] Key type ._llcrypt unregistered [ 4276.820539] alg: No test for adler32 (adler32-zlib) [ 4277.576421] Key type ._llcrypt registered [ 4277.578652] Key type .llcrypt registered [ 4277.662751] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4279.033503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4279.463583] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 4279.792046] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 4279.794993] LNet: Accept secure, port 988 [ 4281.495138] Key type lgssc registered [ 4282.517956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4290.426244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4291.939725] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4292.016961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4297.185818] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4297.229443] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4302.306327] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4306.695692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.293462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4311.726861] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4319.136644] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4325.678372] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4332.182749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4332.649189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4336.979055] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4337.776833] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 4343.922044] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 4344.691808] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4353.516905] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4353.526790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.540287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4354.532246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4354.541067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4354.541521] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4359.336812] Lustre: server umount lustre-OST0000 complete [ 4363.744387] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4363.752941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.772529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4363.778218] Lustre: Skipped 1 previous similar message [ 4368.603648] Lustre: server umount lustre-MDT0000 complete [ 4369.888546] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4369.908146] LustreError: Skipped 1 previous similar message [ 4372.738060] LustreError: 162017:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204031 with bad export cookie 3587582444554267851 [ 4372.743064] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4372.752767] LustreError: 162017:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4373.310033] Lustre: server umount lustre-MDT0001 complete [ 4387.757873] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 4390.427961] Key type lgssc unregistered [ 4390.627159] LNet: 162821:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4390.634263] LNet: Removed LNI 192.168.203.118@tcp [ 4391.503125] Key type .llcrypt unregistered [ 4391.510859] Key type ._llcrypt unregistered [ 4416.056301] alg: No test for adler32 (adler32-zlib) [ 4416.819849] Key type ._llcrypt registered [ 4416.834625] Key type .llcrypt registered [ 4416.931641] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4418.372297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4418.960237] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 4419.278666] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 4419.292092] LNet: Accept secure, port 988 [ 4421.039313] Key type lgssc registered [ 4422.609200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4430.415717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4438.643365] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 18:01:34 (1776204094) [ 4457.618374] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 4468.371212] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4479.049215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4486.259573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4493.293773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4499.710718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4507.238371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4507.320547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.583704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4508.616248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4508.728966] Lustre: lustre-MDT0000: new disk, initializing [ 4508.845931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4508.862971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4513.083827] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4523.347163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4523.428752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4523.585748] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4523.663986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4523.671348] Lustre: Skipped 1 previous similar message [ 4523.758290] Lustre: lustre-MDT0001: new disk, initializing [ 4523.848285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4523.895761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4523.908094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4528.828734] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4538.589352] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4544.343460] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4551.417261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4551.492548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4551.504625] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4551.818848] Lustre: lustre-OST0000: new disk, initializing [ 4551.832146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4551.921350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4556.254738] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4558.928597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4558.940294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4565.152391] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4574.688106] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4574.690772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4574.701534] LustreError: Skipped 1 previous similar message [ 4574.711435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4574.719541] Lustre: Skipped 1 previous similar message [ 4579.048987] Lustre: server umount lustre-OST0000 complete [ 4584.932622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4584.935504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4584.953122] Lustre: Skipped 2 previous similar messages [ 4584.968135] Lustre: Skipped 1 previous similar message [ 4588.457273] Lustre: server umount lustre-MDT0000 complete [ 4590.049227] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.067350] LustreError: Skipped 1 previous similar message [ 4591.075156] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4591.096186] LustreError: Skipped 1 previous similar message [ 4592.007194] LustreError: 168129:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204250 with bad export cookie 3351303651461563970 [ 4592.012437] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4592.021566] LustreError: 168129:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4592.263376] Lustre: server umount lustre-MDT0001 complete [ 4605.993608] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 4609.441563] Key type lgssc unregistered [ 4609.640673] LNet: 171268:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4609.647547] LNet: Removed LNI 192.168.203.118@tcp [ 4610.215904] Key type .llcrypt unregistered [ 4610.217866] Key type ._llcrypt unregistered [ 4631.039263] alg: No test for adler32 (adler32-zlib) [ 4631.794340] Key type ._llcrypt registered [ 4631.796124] Key type .llcrypt registered [ 4631.867253] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4632.772643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4633.222205] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 4633.433471] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 4633.451316] LNet: Accept secure, port 988 [ 4635.159157] Key type lgssc registered [ 4636.233484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4644.002684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4645.479854] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4645.523330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.443476] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4650.976482] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.098330] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4658.272093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4658.578318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4663.462795] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4674.767947] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4682.327430] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4689.654029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4689.955243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4693.226329] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4696.105357] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4700.806653] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4713.810559] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4713.830036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4713.853560] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4713.861473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4713.877025] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 4713.879592] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 4714.589352] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4714.606247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4714.623846] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4714.636873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4714.647986] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 4714.649163] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 4715.307693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4715.318671] Lustre: lustre-OST0000: deleting orphan objects from 0x0:116 to 0x0:161 [ 4715.958680] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4715.963908] Lustre: Skipped 1 previous similar message [ 4715.978913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4715.988594] Lustre: Skipped 1 previous similar message [ 4715.994578] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4715.998841] Lustre: Skipped 1 previous similar message [ 4716.000590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4716.009261] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 4716.009692] Lustre: lustre-OST0000: deleting orphan objects from 0x0:116 to 0x0:193 [ 4716.014919] Lustre: Skipped 1 previous similar message [ 4716.610670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4716.620696] Lustre: lustre-OST0000: deleting orphan objects from 0x0:116 to 0x0:225 [ 4717.195628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4717.208673] Lustre: lustre-OST0000: deleting orphan objects from 0x0:116 to 0x0:257 [ 4717.872466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4717.882392] Lustre: lustre-OST0000: deleting orphan objects from 0x0:116 to 0x0:289 [ 4718.516066] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4718.526080] Lustre: Skipped 3 previous similar messages [ 4718.537202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.544963] Lustre: Skipped 3 previous similar messages [ 4718.547504] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4718.555388] Lustre: Skipped 3 previous similar messages [ 4718.558790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4718.570554] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 4718.572617] Lustre: lustre-OST0000: deleting orphan objects from 0x0:116 to 0x0:321 [ 4718.576038] Lustre: Skipped 3 previous similar messages [ 4719.152880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4719.174675] Lustre: lustre-OST0000: deleting orphan objects from 0x0:116 to 0x0:353 [ 4719.806862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4719.819802] Lustre: lustre-OST0000: deleting orphan objects from 0x0:116 to 0x0:385 [ 4727.782385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4727.785502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4727.793340] Lustre: Skipped 3 previous similar messages [ 4732.902483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4732.908642] Lustre: Skipped 2 previous similar messages [ 4733.101951] Lustre: server umount lustre-OST0000 complete [ 4738.020489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4738.038498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4738.048496] Lustre: Skipped 1 previous similar message [ 4742.880446] Lustre: server umount lustre-MDT0000 complete [ 4743.139745] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4747.179836] LustreError: 172714:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204406 with bad export cookie 7713268491386433316 [ 4747.186482] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4747.197126] LustreError: 172714:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4747.738596] Lustre: server umount lustre-MDT0001 complete [ 4759.477893] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 4762.399396] Key type lgssc unregistered [ 4762.716157] LNet: 177217:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4762.735328] LNet: Removed LNI 192.168.203.118@tcp [ 4763.502384] Key type .llcrypt unregistered [ 4763.505323] Key type ._llcrypt unregistered [ 4785.280151] alg: No test for adler32 (adler32-zlib) [ 4786.054024] Key type ._llcrypt registered [ 4786.056231] Key type .llcrypt registered [ 4786.229949] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4787.333089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4788.002774] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 4788.205580] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 4788.210612] LNet: Accept secure, port 988 [ 4789.879120] Key type lgssc registered [ 4790.862906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4798.118964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4799.645177] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4799.702502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.153650] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4805.095741] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.211598] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4812.494996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4812.819849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4816.931343] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4824.681678] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4830.494435] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4838.601432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4839.138505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4844.694253] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4849.487039] Lustre: lustre-OST0000: deleting orphan objects from 0x0:116 to 0x0:417 [ 4854.340697] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4859.516305] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4861.574778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4861.583076] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4861.587055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4861.598697] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 4861.600499] LustreError: 178701:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 4863.977569] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4863.993864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4864.029699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4864.992581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4865.004840] Lustre: Skipped 1 previous similar message [ 4866.378047] Lustre: lustre-OST0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 4869.644980] Lustre: server umount lustre-OST0000 complete [ 4881.661992] LustreError: 178701:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 4881.665947] LustreError: 178701:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 4902.728281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4903.015482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4906.801321] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 4907.126943] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4907.140391] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 4907.152916] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:66 to 0x280000400:97 [ 4915.713690] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4918.450571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4918.486703] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 4918.506521] Lustre: lustre-OST0000: deleting orphan objects from 0x0:116 to 0x0:449 [ 4921.736549] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 4927.464953] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92c050184800.ost_server_uuid 40 [ 4929.559086] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff92c050184800.ost_server_uuid in FULL state after 0 sec [ 4936.637618] Lustre: server umount lustre-OST0000 complete [ 4940.768512] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4940.781092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.800712] Lustre: Skipped 1 previous similar message [ 4940.814108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4944.354234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4945.954700] Lustre: 177994:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204597/real 1776204597] req@000000008b11dd38 x1862485354244416/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776204604 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4945.988051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4945.992097] Lustre: Skipped 3 previous similar messages [ 4946.611784] Lustre: server umount lustre-MDT0000 complete [ 4950.302190] LustreError: 178660:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204609 with bad export cookie 7375118130262270281 [ 4950.311653] LustreError: 178660:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4950.315947] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4950.734395] Lustre: server umount lustre-MDT0001 complete [ 4962.294187] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 4964.931958] Key type lgssc unregistered [ 4965.211812] LNet: 183974:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4965.230094] LNet: Removed LNI 192.168.203.118@tcp [ 4966.061091] Key type .llcrypt unregistered [ 4966.065822] Key type ._llcrypt unregistered [ 4999.869448] alg: No test for adler32 (adler32-zlib) [ 5000.624476] Key type ._llcrypt registered [ 5000.626147] Key type .llcrypt registered [ 5000.743749] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 5001.869483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5002.710449] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 5002.933875] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 5002.938815] LNet: Accept secure, port 988 [ 5004.623283] Key type lgssc registered [ 5005.972739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5013.934994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5015.262476] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5015.304154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5019.226963] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5020.643280] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5025.764497] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5027.086560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.470810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5032.631745] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5041.793775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5042.250107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.376158] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 5047.439327] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5056.294136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5056.375618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5056.382670] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5056.536295] Lustre: lustre-OST0001: new disk, initializing [ 5056.539821] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5056.635819] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5061.210478] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5062.738135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5062.757146] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5072.433570] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5079.049299] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5098.986713] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5098.993104] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5099.001358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5099.007327] Lustre: Skipped 2 previous similar messages [ 5103.829675] Lustre: server umount lustre-MDT0000 complete [ 5104.100774] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5104.120842] LustreError: Skipped 3 previous similar messages [ 5107.483207] LustreError: 185756:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204766 with bad export cookie 9067065490749577851 [ 5107.483687] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5107.497024] LustreError: 185756:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5107.756812] Lustre: server umount lustre-MDT0001 complete [ 5116.255199] Lustre: 185035:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204768/real 1776204768] req@00000000de0c0aa2 x1862485578659136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776204775 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 5116.284442] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.293157] Lustre: Skipped 1 previous similar message [ 5117.407147] Lustre: 189562:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204770/real 1776204770] req@000000001695ee40 x1862485578659264/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776204776 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5117.517457] Lustre: server umount lustre-OST0000 complete [ 5119.456111] Lustre: 185036:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204771/real 1776204771] req@00000000087441ce x1862485578659328/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776204778 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 5120.772519] Lustre: server umount lustre-OST0001 complete [ 5123.730939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5123.815585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5124.665437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5124.761435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5125.714469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5125.832739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5126.799248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5126.885668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5140.073582] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 5149.764821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5150.498307] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5150.713277] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.779284] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5160.931331] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.939127] LustreError: Skipped 1 previous similar message [ 5165.239069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5165.456375] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5169.777562] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5178.061971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5180.592803] LustreError: 137-5: mylustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5181.642287] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:483 to 0x0:513 [ 5183.271527] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5192.547996] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5192.951181] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5192.956262] Lustre: Skipped 1 previous similar message [ 5194.740279] Lustre: mylustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 5197.662371] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5205.280355] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5208.666605] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5226.978139] LustreError: 11-0: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5226.985305] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5226.998621] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5227.006853] Lustre: Skipped 3 previous similar messages [ 5229.026113] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.033077] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5229.051307] Lustre: Skipped 2 previous similar messages [ 5229.067795] Lustre: Skipped 3 previous similar messages [ 5230.866855] Lustre: server umount mylustre-MDT0000 complete [ 5234.151894] LustreError: 137-5: mylustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5234.172342] LustreError: Skipped 9 previous similar messages [ 5234.542053] LustreError: 194416:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204893 with bad export cookie 9067065490749580581 [ 5234.546573] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5234.558195] LustreError: 194416:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5244.896187] Lustre: 194926:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204897/real 1776204897] req@000000000c52e04b x1862485578690560/t0(0) o39->mylustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776204903 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5244.923943] Lustre: 194926:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5245.020278] Lustre: server umount mylustre-OST0000 complete [ 5245.023332] Lustre: Skipped 1 previous similar message [ 5246.367279] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5252.011403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5252.133975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5253.243419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5253.393714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5254.529534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5254.722393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5255.818160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5255.925050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5271.461853] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 5281.401748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5281.875831] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5286.248537] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5295.837032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.149948] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5308.960367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5311.332609] LustreError: 137-5: tfs-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5311.342815] LustreError: Skipped 3 previous similar messages [ 5312.377697] Lustre: tfs-OST0000: deleting orphan objects from 0x0:483 to 0x0:545 [ 5313.842786] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5322.154287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5324.196732] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 5326.509661] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5334.034118] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5338.107322] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5352.418590] LustreError: 11-0: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5352.438693] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.454448] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 5353.553930] Lustre: server umount tfs-MDT0000 complete [ 5353.555864] Lustre: Skipped 1 previous similar message [ 5356.912995] LustreError: 196570:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205015 with bad export cookie 9067065490749582415 [ 5356.922942] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5365.735744] Lustre: 185036:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776205017/real 1776205017] req@00000000e03cf3c5 x1862485578716544/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776205024 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 5365.755485] Lustre: 185036:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5373.622057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5373.717363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5374.651729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5374.748787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5375.864174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5375.950925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5377.007333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5377.086750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5392.956319] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 5402.976110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.554671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5403.559102] Lustre: Skipped 3 previous similar messages [ 5407.941680] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5417.393913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5422.700130] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5431.844763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5433.149099] Lustre: lustre-OST0000: deleting orphan objects from 0x0:483 to 0x0:577 [ 5437.475353] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5443.561398] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5443.591225] LustreError: Skipped 12 previous similar messages [ 5448.215242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5450.490369] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 5453.004343] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5461.335598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5465.205337] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5477.312560] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 18:18:53 (1776205133) [ 5494.768635] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5494.787932] Lustre: Skipped 4 previous similar messages [ 5494.791905] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5494.792547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5494.817422] Lustre: Skipped 7 previous similar messages [ 5499.747510] Lustre: server umount lustre-MDT0000 complete [ 5499.754672] Lustre: Skipped 3 previous similar messages [ 5503.492955] LustreError: 201872:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205162 with bad export cookie 9067065490749584039 [ 5503.503399] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5503.507568] LustreError: 201872:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5504.994865] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5537.977437] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 5547.723158] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 5557.299917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5564.376563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5571.242303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5577.680583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5584.680938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5584.774190] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 5584.774190] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 5584.774190] [ 5584.796489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 5585.063501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5585.102956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5585.218618] Lustre: lustre-MDT0000: new disk, initializing [ 5585.320669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.327381] Lustre: Skipped 3 previous similar messages [ 5585.359701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5590.188722] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5603.909616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5603.972519] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 5603.972519] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 5603.972519] [ 5603.990256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 5604.140333] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5604.224854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5604.231691] Lustre: Skipped 1 previous similar message [ 5604.338444] Lustre: lustre-MDT0001: new disk, initializing [ 5604.521415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5604.531212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5608.891921] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5620.126253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5620.163587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5620.173764] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5620.331380] Lustre: lustre-OST0000: new disk, initializing [ 5620.335419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5622.080449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5622.095632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5625.525450] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5637.393746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5637.488754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5637.516201] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5637.725729] Lustre: lustre-OST0001: new disk, initializing [ 5637.729452] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5639.624631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5639.646257] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5642.667667] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5655.257345] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 18:21:51 (1776205311) [ 5671.155282] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 18:22:07 (1776205327) [ 5688.799983] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5688.802570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5688.810951] LustreError: Skipped 1 previous similar message [ 5688.818366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5688.828466] Lustre: Skipped 5 previous similar messages [ 5688.850791] Lustre: Skipped 4 previous similar messages [ 5692.032457] Lustre: server umount lustre-OST0000 complete [ 5692.034311] Lustre: Skipped 3 previous similar messages [ 5701.094232] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.112959] LustreError: Skipped 9 previous similar messages [ 5710.303126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5715.037271] LustreError: 209018:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205373 with bad export cookie 9067065490749585565 [ 5715.040210] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.050897] LustreError: 209018:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5733.682633] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 5753.525137] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 5768.180502] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 5770.511810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5778.979660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5786.357956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5793.033577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5800.427070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5808.125973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5808.186332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.459315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5808.484632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5808.577299] Lustre: lustre-MDT0000: new disk, initializing [ 5808.684439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5813.161627] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5823.440495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5823.517933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5823.999687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5829.368805] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5838.720974] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5844.008491] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5850.857126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5850.913144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5850.919744] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5851.167502] Lustre: lustre-OST0000: new disk, initializing [ 5851.170560] Lustre: Skipped 1 previous similar message [ 5851.179893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5851.183556] Lustre: Skipped 2 previous similar messages [ 5851.258856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5851.268388] Lustre: Skipped 5 previous similar messages [ 5852.792688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5852.798016] Lustre: Skipped 1 previous similar message [ 5852.800989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5855.480596] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5863.189658] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5873.637073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5873.643599] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5873.657255] Lustre: Skipped 4 previous similar messages [ 5873.660538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5873.670640] LustreError: Skipped 1 previous similar message [ 5873.678415] Lustre: Skipped 13 previous similar messages [ 5874.800166] Lustre: server umount lustre-OST0000 complete [ 5874.803497] Lustre: Skipped 3 previous similar messages [ 5888.465967] LustreError: 215377:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205547 with bad export cookie 9067065490749586993 [ 5888.475969] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5888.477656] LustreError: 215377:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5901.910761] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 5904.874505] Key type lgssc unregistered [ 5905.176932] LNet: 219238:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5905.186564] LNet: Removed LNI 192.168.203.118@tcp [ 5905.945981] Key type .llcrypt unregistered [ 5905.948712] Key type ._llcrypt unregistered [ 5931.796613] alg: No test for adler32 (adler32-zlib) [ 5932.555466] Key type ._llcrypt registered [ 5932.560127] Key type .llcrypt registered [ 5932.682596] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 5933.897345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5934.511185] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 5934.662616] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 5934.666505] LNet: Accept secure, port 988 [ 5936.319135] Key type lgssc registered [ 5937.523215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5944.984853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.427165] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.474033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.607985] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5951.968825] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5957.092880] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5959.675705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5960.046383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5964.145487] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5972.218507] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5979.033302] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5986.752657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5987.029767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5992.002961] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 5996.306187] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 5999.938920] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6001.349660] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 6005.729150] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6005.734115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6005.744601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6006.755713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6006.764374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.769902] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6009.584080] Lustre: server umount lustre-OST0000 complete [ 6016.479969] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6016.488148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6016.515764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6016.527343] Lustre: Skipped 1 previous similar message [ 6022.113222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6022.130408] Lustre: Skipped 2 previous similar messages [ 6027.233401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6027.250057] Lustre: Skipped 1 previous similar message [ 6028.257866] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6028.412723] Lustre: server umount lustre-MDT0000 complete [ 6032.352452] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6032.378444] LustreError: Skipped 1 previous similar message [ 6032.918270] LustreError: 220752:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205691 with bad export cookie 12671566234236647746 [ 6032.930536] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6032.938152] LustreError: 220752:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6033.248114] Lustre: server umount lustre-MDT0001 complete [ 6044.587510] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 6047.396601] Key type lgssc unregistered [ 6047.723066] LNet: 224482:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6047.733948] LNet: Removed LNI 192.168.203.118@tcp [ 6048.764857] Key type .llcrypt unregistered [ 6048.770553] Key type ._llcrypt unregistered [ 6065.994590] alg: No test for adler32 (adler32-zlib) [ 6066.753696] Key type ._llcrypt registered [ 6066.759034] Key type .llcrypt registered [ 6066.956185] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 6068.262954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6068.936937] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 6069.172656] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 6069.177867] LNet: Accept secure, port 988 [ 6070.891664] Key type lgssc registered [ 6071.858476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6081.029183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6098.591848] LustreError: 225747:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.118@tcp: failed processing log, type 1: rc = -5 [ 6129.311279] LustreError: 225747:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.118@tcp: failed processing log, type 4: rc = -110 [ 6159.073755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6164.039739] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 6172.370889] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6174.685912] Lustre: server umount lustre-OST0000 complete [ 6190.643778] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 6193.689818] Key type lgssc unregistered [ 6193.948421] LNet: 227392:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6193.958408] LNet: Removed LNI 192.168.203.118@tcp [ 6194.694648] Key type .llcrypt unregistered [ 6194.697464] Key type ._llcrypt unregistered [ 6234.067328] alg: No test for adler32 (adler32-zlib) [ 6234.833457] Key type ._llcrypt registered [ 6234.840027] Key type .llcrypt registered [ 6234.960388] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 6236.403575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6236.880753] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 6237.139303] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 6237.143044] LNet: Accept secure, port 988 [ 6238.807099] Key type lgssc registered [ 6239.955771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6253.122325] loop0: detected capacity change from 0 to 204800000 [ 6253.241032] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6253.250562] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6253.468722] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6253.485768] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6253.549372] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6253.599075] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6253.638601] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6253.708823] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6253.810120] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6253.883319] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6254.881253] loop0: detected capacity change from 0 to 204800000 [ 6255.078473] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6255.823638] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6256.856389] loop0: detected capacity change from 0 to 204800000 [ 6257.295296] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6258.330478] loop0: detected capacity change from 0 to 204800000 [ 6258.470899] print_req_error: 12 callbacks suppressed [ 6258.470912] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6258.486527] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6258.590280] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6258.615767] blk_update_request: operation not supported error, dev loop0, sector 58272 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6258.824446] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6259.825063] loop0: detected capacity change from 0 to 204800000 [ 6259.863972] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6263.795617] loop0: detected capacity change from 0 to 204800000 [ 6263.845282] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6267.647419] loop0: detected capacity change from 0 to 204800000 [ 6267.672951] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6271.960981] loop0: detected capacity change from 0 to 204800000 [ 6271.992457] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6275.719238] loop0: detected capacity change from 0 to 204800000 [ 6275.807960] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6275.910434] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6276.017506] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6276.031636] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 6276.052667] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6276.063480] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6276.088520] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 6276.104647] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 6276.114385] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 6276.124143] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6276.146490] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 6276.170880] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 6276.191890] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 6276.210339] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 6276.232787] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 6276.260527] Lustre: 230877:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 6288.867381] Lustre: 231156:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776205941/real 1776205941] req@00000000ec531ada x1862486873604224/t0(0) o251->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776205947 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6288.923093] Lustre: server umount MGS complete [ 6289.997375] loop0: detected capacity change from 0 to 204800000 [ 6290.131878] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6290.920787] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6291.060712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6291.126025] LustreError: 231210:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 6292.270966] loop1: detected capacity change from 0 to 204800000 [ 6292.515876] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6292.616673] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6292.675532] Lustre: 231325:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6292.690932] Lustre: 231325:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6292.711749] Lustre: 231325:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6292.729983] Lustre: 231325:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6292.758490] Lustre: 231325:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 6292.772489] Lustre: 231325:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 6292.787622] Lustre: 231325:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 6292.836586] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6293.074149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6293.128774] LustreError: 231325:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0001: Aborting client recovery [ 6294.213665] loop2: detected capacity change from 0 to 204800000 [ 6294.348333] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6294.433895] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6294.442255] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 6294.488309] Lustre: 231407:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6294.504925] Lustre: 231407:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6294.528178] Lustre: 231407:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6294.540163] Lustre: 231407:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6294.561577] Lustre: 231407:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6294.581961] Lustre: 231407:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6294.589817] Lustre: 231407:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6294.775743] Lustre: Found index 0 for lustre-OST0000, updating log [ 6294.901177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6294.910565] LustreError: 231268:0:(lod_dev.c:437:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 6294.925259] LustreError: 231407:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 6295.918774] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6295.927578] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 6296.065755] loop3: detected capacity change from 0 to 204800000 [ 6296.244750] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6296.345098] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6296.356838] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 6296.384937] Lustre: 231495:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6296.391395] Lustre: 231495:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6296.405523] Lustre: 231495:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6296.419091] Lustre: 231495:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6296.425783] Lustre: 231495:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6296.439824] Lustre: 231495:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6296.457953] Lustre: 231495:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6296.498055] Lustre: Found index 1 for lustre-OST0001, updating log [ 6296.665932] LustreError: 231495:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 6298.553885] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 6298.556213] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 6298.597969] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 6304.917839] LustreError: 231234:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1018us: rc = -2 [ 6305.442435] LustreError: 231233:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 2631us: rc = -2 [ 6305.489384] LustreError: 231233:0:(mdt_identity.c:112:mdt_identity_do_upcall()) Skipped 25 previous similar messages [ 6318.565234] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6318.574193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6318.593085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6322.146025] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6322.147584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6322.167160] Lustre: Skipped 1 previous similar message [ 6322.170200] Lustre: Skipped 1 previous similar message [ 6324.613479] Lustre: server umount lustre-MDT0000 complete [ 6325.441660] LustreError: 231217:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205984 with bad export cookie 17645241741670169906 [ 6325.467692] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6327.265785] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6327.268255] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6327.270473] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6327.270481] Lustre: Skipped 2 previous similar messages [ 6327.281065] LustreError: Skipped 1 previous similar message [ 6327.291627] Lustre: Skipped 2 previous similar messages [ 6330.276130] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.276163] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6330.303377] Lustre: Skipped 2 previous similar messages [ 6331.955207] Lustre: server umount lustre-MDT0001 complete [ 6339.135090] Lustre: server umount lustre-OST0000 complete [ 6346.545152] Lustre: server umount lustre-OST0001 complete [ 6353.482798] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 18:33:30 (1776206010) [ 6382.385848] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 6393.041119] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 6402.194713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6409.827051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6417.024516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6426.176499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6434.649713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6434.698868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6435.089613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6435.126131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6435.265524] Lustre: lustre-MDT0000: new disk, initializing [ 6435.412406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.418988] Lustre: Skipped 1 previous similar message [ 6435.456717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6440.541657] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 6453.517931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6453.640763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6453.830964] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6453.938201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6453.953778] Lustre: Skipped 1 previous similar message [ 6454.124672] Lustre: lustre-MDT0001: new disk, initializing [ 6454.285530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6454.322111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6454.338159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6459.848620] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 6470.935502] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6477.787265] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6484.975642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6485.045264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6485.052522] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6485.265903] Lustre: lustre-OST0000: new disk, initializing [ 6485.278299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6485.403507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6490.128446] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 6492.730077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6492.742609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6500.811353] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6506.334426] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6519.667986] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6525.952543] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 6569.958407] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6569.989110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6570.012515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6570.030709] Lustre: Skipped 3 previous similar messages [ 6580.193673] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6580.199108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6580.227607] Lustre: Skipped 3 previous similar messages [ 6580.399865] Lustre: server umount lustre-OST0000 complete [ 6585.319469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.341695] Lustre: Skipped 2 previous similar messages [ 6594.721605] LustreError: 238569:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206253 with bad export cookie 17645241741670172251 [ 6594.732609] LustreError: 238569:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6594.741300] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6605.420473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6612.025964] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 6622.687134] Lustre: 241117:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206275/real 1776206275] req@0000000094a5cf03 x1862486873673600/t0(0) o251->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776206281 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 6622.740081] Lustre: server umount MGS complete [ 6622.744841] Lustre: Skipped 2 previous similar messages [ 6633.926603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6634.328414] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6634.348109] LustreError: Skipped 3 previous similar messages [ 6634.357465] LustreError: 241755:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 6634.408710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.185315] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 6644.707545] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6644.734088] LustreError: Skipped 1 previous similar message [ 6648.218721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.929612] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 6660.343053] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6665.588728] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6671.697739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6671.979978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6671.988254] Lustre: Skipped 1 previous similar message [ 6676.501476] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 6683.808171] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6715.365450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6715.382617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6715.383780] Lustre: Skipped 1 previous similar message [ 6715.398457] Lustre: Skipped 5 previous similar messages [ 6716.782608] Lustre: server umount lustre-OST0000 complete [ 6725.601062] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6725.604132] LustreError: Skipped 1 previous similar message [ 6730.720855] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6730.740403] LustreError: Skipped 1 previous similar message [ 6732.935542] LustreError: 244538:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206391 with bad export cookie 17645241741670174050 [ 6732.944417] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6732.956602] LustreError: 244538:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6746.331827] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 6748.985696] Key type lgssc unregistered [ 6749.311914] LNet: 245859:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6749.322565] LNet: Removed LNI 192.168.203.118@tcp [ 6750.060896] Key type .llcrypt unregistered [ 6750.065045] Key type ._llcrypt unregistered [ 6788.477608] alg: No test for adler32 (adler32-zlib) [ 6789.236440] Key type ._llcrypt registered [ 6789.238140] Key type .llcrypt registered [ 6789.350268] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 6805.788918] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 6807.529731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6808.330078] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 6808.707125] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 6808.727987] LNet: Accept secure, port 988 [ 6810.464115] Key type lgssc registered [ 6811.883823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6821.220466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6828.551245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6835.089448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6841.415761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6848.059468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6848.135652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6849.364786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6849.391913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6849.488364] Lustre: lustre-MDT0000: new disk, initializing [ 6849.560476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6849.576735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6854.179502] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 6865.344318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6865.422817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6865.518626] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6865.582807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6865.593907] Lustre: Skipped 1 previous similar message [ 6865.737699] Lustre: lustre-MDT0001: new disk, initializing [ 6865.917382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6866.002523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6866.014307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6870.603187] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 6880.341773] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6888.026326] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6896.186749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6896.282848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6896.290675] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6896.530742] Lustre: lustre-OST0000: new disk, initializing [ 6896.539970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6896.685133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6901.781353] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 6902.389330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6902.394525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6911.631875] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6918.103644] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6933.495090] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6937.809104] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 6979.558348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6979.560737] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6979.574024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6979.585460] Lustre: Skipped 1 previous similar message [ 6984.705644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6984.716133] Lustre: Skipped 2 previous similar messages [ 6985.509140] Lustre: server umount lustre-OST0000 complete [ 6988.767852] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6988.775552] LustreError: Skipped 1 previous similar message [ 6988.785512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.800622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6989.793267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.915770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6994.928557] Lustre: Skipped 3 previous similar messages [ 7000.036571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7000.043407] Lustre: Skipped 1 previous similar message [ 7003.103247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7003.246653] Lustre: server umount lustre-MDT0000 complete [ 7005.152276] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7005.166347] LustreError: Skipped 1 previous similar message [ 7006.926865] LustreError: 252864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206665 with bad export cookie 11021140483775346433 [ 7006.943080] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7006.947805] LustreError: 252864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7007.354431] Lustre: server umount lustre-MDT0001 complete [ 7016.250179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7020.379539] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7031.775208] Lustre: 255355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206684/real 1776206684] req@0000000039d876c4 x1862487472388800/t0(0) o251->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776206690 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7031.800351] Lustre: server umount MGS complete [ 7045.703291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7046.335616] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7046.383552] LustreError: 255992:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7046.491381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7051.745826] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7052.437646] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7056.871320] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7061.989537] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7062.212308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7062.632940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7066.679122] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7075.553317] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7083.088639] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7091.299179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7091.697121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7096.103450] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7103.453564] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7127.526248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7127.541094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7127.546507] Lustre: Skipped 1 previous similar message [ 7127.556331] Lustre: Skipped 1 previous similar message [ 7133.055891] Lustre: server umount lustre-OST0000 complete [ 7137.766656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7137.791768] Lustre: Skipped 1 previous similar message [ 7148.001430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7148.008398] Lustre: Skipped 7 previous similar messages [ 7151.071153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7151.180185] Lustre: server umount lustre-MDT0000 complete [ 7153.122601] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7153.137762] LustreError: Skipped 1 previous similar message [ 7154.759235] LustreError: 255959:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206813 with bad export cookie 11021140483775348246 [ 7154.769547] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7154.774265] LustreError: 255959:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7166.443509] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 7169.106552] Key type lgssc unregistered [ 7169.338780] LNet: 260097:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7169.363542] LNet: Removed LNI 192.168.203.118@tcp [ 7170.100872] Key type .llcrypt unregistered [ 7170.102821] Key type ._llcrypt unregistered [ 7196.155408] alg: No test for adler32 (adler32-zlib) [ 7196.910474] Key type ._llcrypt registered [ 7196.913481] Key type .llcrypt registered [ 7197.053773] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7198.289463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7198.704431] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 7198.919274] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 7198.932078] LNet: Accept secure, port 988 [ 7200.623159] Key type lgssc registered [ 7201.459299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7208.961964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7210.478247] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7210.540901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7215.587045] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7215.861950] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7220.733887] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.680095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7225.201924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7229.505190] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7239.531399] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7245.617178] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7251.962482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7252.355386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.936068] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7264.492615] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7272.382139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7272.460473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7272.465532] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7272.551885] Lustre: lustre-OST0001: new disk, initializing [ 7272.554974] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7272.700465] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7277.729156] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7281.722097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7281.733214] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7281.782169] LustreError: 265007:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7282.861534] LustreError: 265013:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7282.867223] LustreError: 265013:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 7283.872693] LustreError: 265007:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7285.921066] LustreError: 265007:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7285.950940] LustreError: 265007:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 7290.014843] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7290.081765] LustreError: 265013:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7290.088875] LustreError: 265013:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [ 7290.165618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7297.933429] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92c059b63800.ost_server_uuid 40 [ 7298.347132] LustreError: 265013:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7298.367460] LustreError: 265013:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [ 7299.621835] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff92c059b63800.ost_server_uuid in FULL state after 0 sec [ 7306.000904] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92c059b63800.ost_server_uuid 40 [ 7307.721686] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92c059b63800.ost_server_uuid in IDLE state after 0 sec [ 7323.117490] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7323.138876] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7323.155370] Lustre: Skipped 1 previous similar message [ 7326.017458] Lustre: lustre-OST0001: Not available for connect from 192.168.203.18@tcp (stopping) [ 7328.227891] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7328.238963] Lustre: Skipped 1 previous similar message [ 7329.015089] Lustre: server umount lustre-OST0001 complete [ 7331.121626] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7338.464460] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7338.464613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7338.478647] LustreError: Skipped 1 previous similar message [ 7338.479294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7338.514629] Lustre: Skipped 2 previous similar messages [ 7338.521825] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7338.537940] LustreError: Skipped 3 previous similar messages [ 7341.190189] Lustre: server umount lustre-OST0000 complete [ 7348.192512] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7348.200714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7348.212800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7348.215465] Lustre: Skipped 1 previous similar message [ 7350.904669] Lustre: server umount lustre-MDT0000 complete [ 7353.829817] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7353.851629] LustreError: Skipped 1 previous similar message [ 7354.814683] LustreError: 261646:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207013 with bad export cookie 3735784237358339393 [ 7354.817937] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7354.825933] LustreError: 261646:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7355.164517] Lustre: server umount lustre-MDT0001 complete [ 7366.867410] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 7369.211047] Key type lgssc unregistered [ 7369.458860] LNet: 267328:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7369.469986] LNet: Removed LNI 192.168.203.118@tcp [ 7370.275477] Key type .llcrypt unregistered [ 7370.278076] Key type ._llcrypt unregistered [ 7413.415484] alg: No test for adler32 (adler32-zlib) [ 7414.171317] Key type ._llcrypt registered [ 7414.175715] Key type .llcrypt registered [ 7414.320150] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7415.191960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7415.721953] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 7415.890982] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 7415.895615] LNet: Accept secure, port 988 [ 7417.519115] Key type lgssc registered [ 7418.627556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7427.308693] loop0: detected capacity change from 0 to 1099511627776 [ 7428.020918] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 7428.033381] *** VALIDATE xfs *** [ 7428.046371] XFS (loop0): Mounting V5 Filesystem [ 7428.097160] XFS (loop0): Ending clean mount [ 7429.621543] loop1: detected capacity change from 0 to 18691697672192 [ 7524.936498] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7541.790227] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 7550.950461] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7560.736714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7569.495749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7578.303361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7585.997398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7594.161324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7594.225246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7595.510854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7595.588548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7595.677664] Lustre: lustre-MDT0000: new disk, initializing [ 7595.754593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7595.771353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7600.130405] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7611.556381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7611.630302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7611.715374] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7611.774242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7611.788334] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7611.800791] Lustre: Skipped 1 previous similar message [ 7611.917645] Lustre: lustre-MDT0001: new disk, initializing [ 7612.004459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7612.046061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7612.067450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7616.529777] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7627.138940] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7633.575689] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7640.424250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7640.470472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7640.476835] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7640.644115] Lustre: lustre-OST0000: new disk, initializing [ 7640.646606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7640.693786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.328156] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7646.862490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7646.870301] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7656.130210] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7661.456955] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7661.732629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7666.768790] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7667.072202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7672.807927] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7672.808651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7672.839928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7672.842801] Lustre: Skipped 1 previous similar message [ 7675.615793] Lustre: server umount lustre-OST0000 complete [ 7683.040658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.042863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7683.066628] Lustre: Skipped 2 previous similar messages [ 7687.405526] Lustre: server umount lustre-MDT0000 complete [ 7688.163257] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.182774] LustreError: Skipped 1 previous similar message [ 7689.187290] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7689.200842] LustreError: Skipped 1 previous similar message [ 7690.918686] LustreError: 272301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207349 with bad export cookie 194998934169211437 [ 7690.930303] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7690.933600] LustreError: 272301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7691.307555] Lustre: server umount lustre-MDT0001 complete [ 7699.717725] XFS (loop0): Unmounting Filesystem [ 7704.223742] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 18:56:00 (1776207360) [ 7710.891708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7711.423853] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.492249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7716.318236] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7716.843059] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7721.959954] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7725.790381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7726.338099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7731.634910] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7739.659632] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7745.971478] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7753.347042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7753.754273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7758.808564] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7767.581444] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7777.772702] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7777.784674] LustreError: Skipped 1 previous similar message [ 7777.791833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7777.806775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7777.812636] Lustre: Skipped 1 previous similar message [ 7778.784183] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7783.758815] Lustre: server umount lustre-OST0000 complete [ 7788.003609] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7788.012100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7788.027187] Lustre: Skipped 1 previous similar message [ 7788.039903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7788.049023] Lustre: Skipped 2 previous similar messages [ 7793.432595] Lustre: server umount lustre-MDT0000 complete [ 7794.147554] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7794.188743] LustreError: Skipped 1 previous similar message [ 7798.394541] LustreError: 276990:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207457 with bad export cookie 194998934169212291 [ 7798.396629] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7798.402370] LustreError: 276990:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7810.753453] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 7813.687469] Key type lgssc unregistered [ 7813.967191] LNet: 280762:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7813.978617] LNet: Removed LNI 192.168.203.118@tcp [ 7814.625192] Key type .llcrypt unregistered [ 7814.629283] Key type ._llcrypt unregistered [ 7842.176903] alg: No test for adler32 (adler32-zlib) [ 7842.933933] Key type ._llcrypt registered [ 7842.938503] Key type .llcrypt registered [ 7843.053596] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7844.002582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7844.588434] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 7844.782499] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 7844.790837] LNet: Accept secure, port 988 [ 7846.503292] Key type lgssc registered [ 7847.529857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7855.250737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7856.626668] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7856.681265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7861.018105] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7861.731279] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7866.854096] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7869.148458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7869.569555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7873.966819] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7882.177925] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7888.102886] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7894.187356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7894.511524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7898.956339] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7906.382380] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7917.027446] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7917.046492] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7917.057665] Lustre: Skipped 1 previous similar message [ 7918.418607] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [ 7919.664318] Lustre: server umount lustre-MDT0001 complete [ 7922.168653] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7922.186988] LustreError: Skipped 1 previous similar message [ 7923.062383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7927.265269] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7927.277745] LustreError: Skipped 2 previous similar messages [ 7932.804029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7933.017116] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7933.025302] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7933.119968] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7933.563539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7938.679607] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 7939.048375] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7939.064548] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 7939.075809] Lustre: Skipped 1 previous similar message [ 7940.090612] LustreError: 286045:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 7940.101671] LustreError: 286045:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7940.113367] LustreError: 286045:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7940.133711] LustreError: 282389:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7940.192047] LustreError: 286052:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.118@tcp: cfg command failed: rc = -17 [ 7940.224162] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 7940.224162] [ 8246.108523] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [ 8246.242213] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8246.249627] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.281568] Lustre: Skipped 1 previous similar message [ 8248.302400] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8248.316840] Lustre: Skipped 1 previous similar message [ 8248.343249] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.265176] Lustre: server umount lustre-MDT0001 complete [ 8251.175622] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8251.216650] LustreError: Skipped 3 previous similar messages [ 8254.246894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8262.118897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8262.217716] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8262.228668] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8262.297228] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8262.661101] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 8267.016495] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 8267.766864] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8267.785328] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 8267.798201] Lustre: Skipped 1 previous similar message [ 8272.374625] LustreError: 294989:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 8272.385703] LustreError: 294989:0:(genops.c:522:class_register_device()) Skipped 1 previous similar message [ 8272.391664] LustreError: 294989:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8272.400302] LustreError: 294989:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8272.406717] LustreError: 282389:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 8272.410818] LustreError: 282389:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 8272.461860] LustreError: 294992:0:(obd_config.c:680:class_setup()) Device 8 already setup (type osp) [ 8272.467425] LustreError: 294992:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.118@tcp: cfg command failed: rc = -17 [ 8272.477190] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.118@tcp [ 8272.477190] [ 8573.776066] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [ 8573.783216] Lustre: Skipped 1 previous similar message [ 8574.944953] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8574.960948] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8579.725075] Lustre: server umount lustre-MDT0001 complete [ 8583.141262] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8583.160358] LustreError: Skipped 7 previous similar messages [ 8583.181059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8593.559887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8593.726969] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8593.735214] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8593.802106] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8594.157364] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 8599.535252] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8599.560627] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 8599.570445] Lustre: Skipped 1 previous similar message [ 8600.464280] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 8601.598512] LustreError: 303545:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 8601.609741] LustreError: 303545:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8601.624983] LustreError: 303545:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8601.636582] LustreError: 282389:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 8601.649713] LustreError: 282389:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 8601.758046] LustreError: 303564:0:(lod_lov.c:228:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 8601.797340] LustreError: 303564:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.118@tcp: cfg command failed: rc = -114 [ 8601.809781] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 8601.809781] [ 8922.083214] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8922.085754] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8922.086108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8922.086112] Lustre: Skipped 4 previous similar messages [ 8922.124616] Lustre: Skipped 2 previous similar messages [ 8927.984908] Lustre: server umount lustre-MDT0000 complete [ 8932.323680] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8932.355660] LustreError: Skipped 8 previous similar messages [ 8932.861233] LustreError: 285661:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208591 with bad export cookie 3715999843562433527 [ 8932.862993] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8932.877548] LustreError: 285661:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8937.450683] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8937.470116] Lustre: Skipped 1 previous similar message [ 8938.534709] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8938.543557] Lustre: Skipped 6 previous similar messages [ 8939.389895] Lustre: server umount lustre-MDT0001 complete [ 8955.364737] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 8965.981943] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8978.006894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8984.338298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8991.107695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8999.648043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9009.519620] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 19:17:45 (1776208665) [ 9018.491537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9018.582393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9018.862416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9018.907117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9019.018299] Lustre: lustre-MDT0000: new disk, initializing [ 9019.151022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9019.173047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9025.776467] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 9037.702258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9037.771494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9037.893551] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9037.951579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9037.955519] Lustre: Skipped 1 previous similar message [ 9038.059061] Lustre: lustre-MDT0001: new disk, initializing [ 9038.148105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9038.188038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9038.196858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9042.937965] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 9054.596267] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9060.948605] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9067.984102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9068.065847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9068.079489] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9068.440888] Lustre: lustre-OST0000: new disk, initializing [ 9068.451585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9068.624643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9073.731952] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all 8 [ 9077.341132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9077.358154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9083.590728] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9093.094929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9093.101940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9093.110311] Lustre: Skipped 2 previous similar messages [ 9093.115135] Lustre: Skipped 1 previous similar message [ 9093.394917] Lustre: server umount lustre-MDT0000 complete [ 9093.397194] Lustre: Skipped 1 previous similar message [ 9098.028906] LustreError: 316529:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208756 with bad export cookie 3715999843562435445 [ 9098.039517] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9098.048758] LustreError: 316529:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9098.048975] LustreError: 315310:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 9098.049289] LustreError: 315310:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e284574f x1862488558860864/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' [ 9098.049405] LustreError: 315310:0:(llog_cat.c:739:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 9098.049415] LustreError: 315310:0:(llog_cat.c:775:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 9098.208395] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9098.219913] LustreError: Skipped 2 previous similar messages [ 9134.716644] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 9143.147428] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 9153.511529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9160.971768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9167.795151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9174.915627] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9176.992025] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8936 sec ========== 19:20:33 (1776208833) [ 9222.730170] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 9225.193298] Key type lgssc unregistered [ 9225.452313] LNet: 322807:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9225.470330] LNet: Removed LNI 192.168.203.118@tcp [ 9226.108537] Key type .llcrypt unregistered [ 9226.124404] Key type ._llcrypt unregistered