[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 430948205 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003098] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008595] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010006] pid_max: default: 32768 minimum: 301 [ 0.011131] LSM: Security Framework initializing [ 0.012031] Yama: becoming mindful. [ 0.013023] SELinux: Initializing. [ 0.014039] *** VALIDATE selinux *** [ 0.021098] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026019] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027086] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028064] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029062] *** VALIDATE tmpfs *** [ 0.030332] *** VALIDATE proc *** [ 0.031168] *** VALIDATE cgroup *** [ 0.032017] *** VALIDATE cgroup2 *** [ 0.033104] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034116] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036017] Spectre V2 : User space: Vulnerable [ 0.037003] Speculative Store Bypass: Vulnerable [ 0.040064] debug: unmapping init [mem 0xffffffffa8859000-0xffffffffa8860fff] [ 0.042191] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043492] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044009] ... version: 2 [ 0.044900] ... bit width: 48 [ 0.045007] ... generic registers: 4 [ 0.045959] ... value mask: 0000ffffffffffff [ 0.046006] ... max period: 00007fffffffffff [ 0.047006] ... fixed-purpose events: 3 [ 0.047952] ... event mask: 000000070000000f [ 0.048231] rcu: Hierarchical SRCU implementation. [ 0.050275] smp: Bringing up secondary CPUs ... [ 0.051470] x86: Booting SMP configuration: [ 0.052011] .... node #0, CPUs: #1 #2 #3 [ 0.055208] smp: Brought up 1 node, 4 CPUs [ 0.056934] smpboot: Max logical packages: 1 [ 0.057008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.235373] node 0 deferred pages initialised in 177ms [ 0.238007] devtmpfs: initialized [ 0.238880] x86/mm: Memory block size: 128MB [ 0.240249] gcov: version magic: 0x41383552 [ 0.241569] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.242044] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243163] pinctrl core: initialized pinctrl subsystem [ 0.244115] [ 0.244413] ************************************************************* [ 0.245007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.246005] ** ** [ 0.247014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.248006] ** ** [ 0.249006] ** This means that this kernel is built to expose internal ** [ 0.250006] ** IOMMU data structures, which may compromise security on ** [ 0.251007] ** your system. ** [ 0.252006] ** ** [ 0.253006] ** If you see this message and you are not debugging the ** [ 0.254007] ** kernel, report this immediately to your vendor! ** [ 0.255005] ** ** [ 0.256006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.257016] ************************************************************* [ 0.258804] NET: Registered protocol family 16 [ 0.259750] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.260090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.261127] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.263395] cpuidle: using governor menu [ 0.266134] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.268564] PCI: Using configuration type 1 for base access [ 0.270141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.278054] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.279019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.280120] cryptd: max_cpu_qlen set to 1000 [ 0.283181] ACPI: Added _OSI(Module Device) [ 0.284010] ACPI: Added _OSI(Processor Device) [ 0.286010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.287008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.291980] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.298677] ACPI: Interpreter enabled [ 0.300049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.301010] ACPI: Using IOAPIC for interrupt routing [ 0.303096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.307430] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.318762] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.321023] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.323012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.326053] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.332000] acpiphp: Slot [2] registered [ 0.333111] acpiphp: Slot [5] registered [ 0.335106] acpiphp: Slot [6] registered [ 0.337072] acpiphp: Slot [7] registered [ 0.338070] acpiphp: Slot [8] registered [ 0.340078] acpiphp: Slot [9] registered [ 0.341071] acpiphp: Slot [10] registered [ 0.341838] acpiphp: Slot [3] registered [ 0.343045] acpiphp: Slot [4] registered [ 0.343982] acpiphp: Slot [11] registered [ 0.345074] acpiphp: Slot [12] registered [ 0.346070] acpiphp: Slot [13] registered [ 0.348143] acpiphp: Slot [14] registered [ 0.350066] acpiphp: Slot [15] registered [ 0.351085] acpiphp: Slot [16] registered [ 0.352063] acpiphp: Slot [17] registered [ 0.352970] acpiphp: Slot [18] registered [ 0.354047] acpiphp: Slot [19] registered [ 0.354918] acpiphp: Slot [20] registered [ 0.356063] acpiphp: Slot [21] registered [ 0.357045] acpiphp: Slot [22] registered [ 0.358097] acpiphp: Slot [23] registered [ 0.360095] acpiphp: Slot [24] registered [ 0.361072] acpiphp: Slot [25] registered [ 0.363088] acpiphp: Slot [26] registered [ 0.365087] acpiphp: Slot [27] registered [ 0.366083] acpiphp: Slot [28] registered [ 0.368083] acpiphp: Slot [29] registered [ 0.370092] acpiphp: Slot [30] registered [ 0.371068] acpiphp: Slot [31] registered [ 0.373078] PCI host bridge to bus 0000:00 [ 0.375013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.377012] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.379012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.382011] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.385012] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.388014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.390224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.393206] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.397390] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.407797] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.412106] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.416013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.418017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.421090] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.423757] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.426808] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.430029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.432811] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.438015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.451016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.457013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.463278] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.467008] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.472007] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.483015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.490982] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.497013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.503010] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.517012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.526619] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.533009] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.536010] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.547013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.553208] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.558008] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.564013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.574013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.581803] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.588011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.595012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.608018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.619563] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.625010] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.631008] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.642017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.653767] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.656295] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.658430] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.660297] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.662155] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.667110] iommu: Default domain type: Passthrough [ 0.670673] SCSI subsystem initialized [ 0.672125] ACPI: bus type USB registered [ 0.674151] usbcore: registered new interface driver usbfs [ 0.676065] usbcore: registered new interface driver hub [ 0.678049] usbcore: registered new device driver usb [ 0.680183] pps_core: LinuxPPS API ver. 1 registered [ 0.682010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.686066] PTP clock support registered [ 0.689055] EDAC MC: Ver: 3.0.0 [ 0.691165] PCI: Using ACPI for IRQ routing [ 0.692862] NetLabel: Initializing [ 0.694009] NetLabel: domain hash size = 128 [ 0.696009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.698059] NetLabel: unlabeled traffic allowed by default [ 0.700100] vgaarb: loaded [ 0.701307] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.703010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.710083] clocksource: Switched to clocksource kvm-clock [ 0.800505] VFS: Disk quotas dquot_6.6.0 [ 0.802045] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.804528] *** VALIDATE ramfs *** [ 0.805759] *** VALIDATE hugetlbfs *** [ 0.807365] pnp: PnP ACPI init [ 0.809761] pnp: PnP ACPI: found 6 devices [ 0.824495] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.827819] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.830015] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.832216] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.834595] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.837055] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.839883] NET: Registered protocol family 2 [ 0.842368] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.847136] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.849443] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.853294] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.855554] TCP: Hash tables configured (established 65536 bind 65536) [ 0.857457] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.859348] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.860966] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.862711] NET: Registered protocol family 1 [ 0.864341] RPC: Registered named UNIX socket transport module. [ 0.866317] RPC: Registered udp transport module. [ 0.867253] RPC: Registered tcp transport module. [ 0.868343] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.869471] NET: Registered protocol family 44 [ 0.870507] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.872099] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.874172] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.876456] PCI: CLS 0 bytes, default 64 [ 0.878037] Unpacking initramfs... [ 2.283810] debug: unmapping init [mem 0xffff9b6a3cc54000-0xffff9b6a3ffbffff] [ 2.291393] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.293737] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.297956] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.901674] Initialise system trusted keyrings [ 2.903321] Key type blacklist registered [ 2.905067] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.913250] zbud: loaded [ 2.915827] *** VALIDATE nfs *** [ 2.916849] *** VALIDATE nfs4 *** [ 2.918121] pstore: using deflate compression [ 2.920283] Platform Keyring initialized [ 2.991291] NET: Registered protocol family 38 [ 2.994313] Key type asymmetric registered [ 2.996453] Asymmetric key parser 'x509' registered [ 2.998739] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.002334] io scheduler mq-deadline registered [ 3.003820] io scheduler kyber registered [ 3.005428] io scheduler bfq registered [ 3.007250] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.010224] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.013027] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.015878] ACPI: Power Button [PWRF] [ 3.020903] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.026771] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.038509] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.044891] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.059780] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.085573] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.112581] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.116963] Non-volatile memory driver v1.3 [ 3.117909] Linux agpgart interface v0.103 [ 3.152543] virtio_blk virtio1: [vda] 67992 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.156454] vda: detected capacity change from 0 to 34811904 [ 3.171105] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.172863] vdb: detected capacity change from 0 to 1073741824 [ 3.188742] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.191348] vdc: detected capacity change from 0 to 2621440000 [ 3.204137] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.206289] vdd: detected capacity change from 0 to 2621440000 [ 3.217813] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.220290] vde: detected capacity change from 0 to 4294967296 [ 3.231027] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.232704] vdf: detected capacity change from 0 to 4294967296 [ 3.238183] libphy: Fixed MDIO Bus: probed [ 3.242567] usbcore: registered new interface driver usbserial_generic [ 3.244974] usbserial: USB Serial support registered for generic [ 3.246913] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.250645] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.252351] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.255184] mousedev: PS/2 mouse device common for all mice [ 3.257673] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.260320] rtc_cmos 00:05: RTC can wake from S4 [ 3.261957] rtc_cmos 00:05: registered as rtc0 [ 3.264268] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.264311] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.266457] intel_pstate: CPU model not supported [ 3.271584] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.283226] hid: raw HID events driver (C) Jiri Kosina [ 3.285568] usbcore: registered new interface driver usbhid [ 3.286741] usbhid: USB HID core driver [ 3.288387] drop_monitor: Initializing network drop monitor service [ 3.290529] Initializing XFRM netlink socket [ 3.292230] NET: Registered protocol family 10 [ 3.297477] Segment Routing with IPv6 [ 3.298642] NET: Registered protocol family 17 [ 3.300592] mpls_gso: MPLS GSO support [ 3.307035] RAS: Correctable Errors collector initialized. [ 3.308526] AVX version of gcm_enc/dec engaged. [ 3.310112] AES CTR mode by8 optimization enabled [ 3.383589] sched_clock: Marking stable (3383558887, 0)->(4239423628, -855864741) [ 3.386321] registered taskstats version 1 [ 3.387921] Loading compiled-in X.509 certificates [ 3.390444] zswap: loaded using pool lzo/zbud [ 3.413248] Key type big_key registered [ 3.426895] Key type encrypted registered [ 3.428555] ima: No TPM chip found, activating TPM-bypass! [ 3.430330] ima: Allocated hash algorithm: sha1 [ 3.431350] ima: No architecture policies found [ 3.432371] evm: Initialising EVM extended attributes: [ 3.433850] evm: security.selinux [ 3.435130] evm: security.ima [ 3.435695] evm: security.capability [ 3.436391] evm: HMAC attrs: 0x1 [ 3.438394] rtc_cmos 00:05: setting system clock to 2026-02-28 18:51:30 UTC (1772304690) [ 3.444554] debug: unmapping init [mem 0xffffffffa9803000-0xffffffffa99fffff] [ 3.447535] debug: unmapping init [mem 0xffffffffa8582000-0xffffffffa8858fff] [ 3.456146] Write protecting the kernel read-only data: 28672k [ 3.459118] debug: unmapping init [mem 0xffffffffa6c03000-0xffffffffa6dfffff] [ 3.464240] debug: unmapping init [mem 0xffffffffa7514000-0xffffffffa75fffff] [ 3.499078] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.507967] systemd[1]: Detected virtualization kvm. [ 3.510134] systemd[1]: Detected architecture x86-64. [ 3.512201] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.544399] systemd[1]: No hostname configured. [ 3.546069] systemd[1]: Set hostname to . [ 3.548238] random: systemd: uninitialized urandom read (16 bytes read) [ 3.551334] systemd[1]: Initializing machine ID from random generator. [ 3.612139] random: ln: uninitialized urandom read (6 bytes read) [ 3.683418] random: systemd: uninitialized urandom read (16 bytes read) [ 3.685419] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.688903] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.692561] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.272305] device-mapper: uevent: version 1.0.3 [ 4.273510] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.030222] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.043686] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 5.400782] scsi host0: ata_piix [ 5.466097] scsi host1: ata_piix [ 5.468847] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.473136] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.953981] random: crng init done [ 9.956006] random: 7 urandom warning(s) missed due to ratelimiting [ 10.123377] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.371909] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.610789] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.965189] SELinux: Disabled at runtime. [ 13.029794] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.037458] systemd[1]: Detected virtualization kvm. [ 13.038869] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.640908] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.643867] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.648476] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.651609] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.653527] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.662157] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.669923] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ 13.708368] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 14.286315] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.708047] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.739254] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.096122] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.123958] EDAC sbridge: Ver: 1.1.2 [ 16.786526] Key type dns_resolver registered [ 17.130666] NFS: Registering the id_resolver key type [ 17.133577] Key type id_resolver registered [ 17.135176] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg238-server login: [ 38.780972] libcfs: loading out-of-tree module taints kernel. [ 38.796348] alg: No test for adler32 (adler32-zlib) [ 39.548296] Key type ._llcrypt registered [ 39.549333] Key type .llcrypt registered [ 39.579723] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 46.230170] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 46.801317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 47.053446] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 47.298722] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 47.300383] LNet: Accept secure, port 988 [ 48.903150] Key type lgssc registered [ 49.404576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.258937] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 56.417171] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 59.847982] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 62.289279] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 64.760856] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 69.628388] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 73.757243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 73.782846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 74.862588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 74.871385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 74.906848] Lustre: lustre-MDT0000: new disk, initializing [ 74.927501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 74.933571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 76.311153] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 81.322545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.355618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 81.384845] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 81.393624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 81.395372] Lustre: Skipped 1 previous similar message [ 81.433420] Lustre: lustre-MDT0001: new disk, initializing [ 81.453934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 81.464202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 81.468577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 82.793485] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 87.725115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.748678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 87.751301] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 87.834067] Lustre: lustre-OST0000: new disk, initializing [ 87.836274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 87.851940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 89.259231] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 90.988259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 90.990885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 94.284522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.311853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 94.316074] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 94.352080] Lustre: lustre-OST0001: new disk, initializing [ 94.353859] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 94.368425] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 95.708558] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 100.329686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 100.332617] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 101.070559] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 103.646649] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 109.921976] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing check_logdir /tmp/testlogs/ [ 111.502707] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing yml_node [ 113.052849] Lustre: DEBUG MARKER: Client: 2.15.8.1 [ 113.884058] Lustre: DEBUG MARKER: MDS: 2.15.8.1 [ 114.691847] Lustre: DEBUG MARKER: OSS: 2.15.8.1 [ 115.184337] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Feb 28 13:53:22 EST 2026 [ 117.760899] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 118.264406] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 131.039797] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 131.040302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 131.046459] Lustre: Skipped 2 previous similar messages [ 131.048751] Lustre: Skipped 1 previous similar message [ 136.160018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 136.163462] Lustre: Skipped 5 previous similar messages [ 141.279772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 141.282378] Lustre: Skipped 3 previous similar messages [ 144.351136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 144.386380] Lustre: server umount lustre-MDT0000 complete [ 145.526554] LustreError: 7232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772304832 with bad export cookie 8347396495197636609 [ 145.528102] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 145.529952] LustreError: 7232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 146.399595] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 146.399722] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 146.400028] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 146.403533] LustreError: Skipped 1 previous similar message [ 146.409643] Lustre: Skipped 2 previous similar messages [ 150.431463] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 150.431695] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 150.434164] Lustre: Skipped 2 previous similar messages [ 150.438272] LustreError: Skipped 1 previous similar message [ 151.456340] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 151.459701] LustreError: Skipped 1 previous similar message [ 151.727590] Lustre: server umount lustre-MDT0001 complete [ 167.391109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 167.403651] Lustre: server umount lustre-OST0000 complete [ 174.670812] Lustre: server umount lustre-OST0001 complete [ 176.658835] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 179.592065] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 183.253850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.440908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.372400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.319580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.532279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.559930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.645954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 191.657644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 191.698360] Lustre: lustre-MDT0000: new disk, initializing [ 191.719040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.724931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 193.090225] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 197.260948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 197.287569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.320430] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 197.411022] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 198.768449] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 202.422342] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 204.339832] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 206.518021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.544648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.548028] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 206.632357] Lustre: lustre-OST0000: new disk, initializing [ 206.634436] Lustre: Skipped 1 previous similar message [ 206.636744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 206.640173] Lustre: Skipped 2 previous similar messages [ 206.663524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.665605] Lustre: Skipped 1 previous similar message [ 207.990733] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 208.437593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.440450] Lustre: Skipped 1 previous similar message [ 208.442293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.711783] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 213.160939] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 213.241382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 214.663372] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 214.745169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 218.595411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 218.596530] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 218.596915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 218.596920] Lustre: Skipped 1 previous similar message [ 218.599589] Lustre: Skipped 1 previous similar message [ 221.493865] Lustre: server umount lustre-OST0000 complete [ 223.199952] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 223.202570] LustreError: Skipped 1 previous similar message [ 223.204180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.205086] LustreError: 14306:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772304917 with bad export cookie 8347396495197638051 [ 230.208918] LustreError: 14306:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 230.209064] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 230.354791] Lustre: server umount lustre-MDT0001 complete [ 230.356301] Lustre: Skipped 1 previous similar message [ 231.818167] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 13:55:18 (1772304918) [ 234.056938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 234.197203] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 234.218903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.494730] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 238.414371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 239.834153] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 242.277562] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 244.064840] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 246.220366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.763060] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 250.232151] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 255.895411] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 259.042629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.044206] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 259.044514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 259.044522] Lustre: Skipped 5 previous similar messages [ 259.046657] Lustre: Skipped 3 previous similar messages [ 264.966991] Lustre: server umount lustre-MDT0000 complete [ 266.162105] LustreError: 18693:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772304953 with bad export cookie 8347396495197638905 [ 266.164801] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 266.165908] LustreError: 18693:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 273.887188] Lustre: 21819:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772304954/real 1772304954] req@00000000acb8a34e x1858396208822400/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772304960 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 280.389945] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 283.707578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 283.844323] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 283.858434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 283.860261] Lustre: Skipped 2 previous similar messages [ 285.173365] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 287.823619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 289.276653] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 291.962054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 293.467614] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 296.147675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 296.173272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 296.178180] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 296.230671] Lustre: lustre-OST0001: new disk, initializing [ 296.232959] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 297.658956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 297.664307] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 297.674880] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 302.013869] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 303.191404] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 313.824108] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 313.826297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.829790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.831582] Lustre: Skipped 5 previous similar messages [ 315.023903] Lustre: server umount lustre-MDT0000 complete [ 315.025605] Lustre: Skipped 2 previous similar messages [ 315.361083] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 315.365253] LustreError: Skipped 2 previous similar messages [ 316.208273] LustreError: 25524:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305003 with bad export cookie 8347396495197639752 [ 316.211432] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.213624] LustreError: 25524:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 325.087099] Lustre: 27060:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305006/real 1772305006] req@00000000cb61ebb0 x1858396208846272/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772305012 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 332.767172] Lustre: 27261:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305013/real 1772305013] req@00000000cb61ebb0 x1858396208846784/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772305019 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 338.150460] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 341.377948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 341.515708] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.523756] LustreError: Skipped 1 previous similar message [ 342.969445] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 345.712187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.202288] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 350.075591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 350.212896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 350.216335] Lustre: Skipped 5 previous similar messages [ 351.678812] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 354.510769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 356.088062] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 359.133301] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 360.366498] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 360.369139] Lustre: Skipped 1 previous similar message [ 367.075668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.079599] Lustre: Skipped 5 previous similar messages [ 374.331470] LustreError: 28475:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305061 with bad export cookie 8347396495197640998 [ 374.334626] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 374.335415] LustreError: 28475:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 381.919084] Lustre: 32093:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305062/real 1772305062] req@00000000096121a4 x1858396208864832/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772305068 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 381.940547] Lustre: server umount lustre-OST0000 complete [ 381.941829] Lustre: Skipped 5 previous similar messages [ 389.599133] Lustre: 32294:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305070/real 1772305070] req@00000000683a47c8 x1858396208865344/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772305076 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 392.623263] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 13:57:59 (1772305079) [ 401.475459] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 404.972740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.126684] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 405.131465] LustreError: Skipped 1 previous similar message [ 406.581582] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 409.468759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.021405] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 414.271162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 416.336764] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 419.851558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 421.416389] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 424.613632] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 426.042130] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 435.168087] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 435.170642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.173731] Lustre: Skipped 1 previous similar message [ 435.175593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 435.177457] Lustre: Skipped 8 previous similar messages [ 440.035781] LustreError: 36670:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305127 with bad export cookie 8347396495197642069 [ 440.042353] LustreError: 36670:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 440.042504] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.943150] Lustre: 37879:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305128/real 1772305128] req@000000000838835c x1858396208883840/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772305134 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 451.011888] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 13:58:58 (1772305138) [ 455.988924] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 459.414352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 461.125420] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 464.786829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.286746] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 468.966180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 470.116003] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.120994] LustreError: Skipped 2 previous similar messages [ 470.382108] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 473.130548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 474.506614] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 477.694116] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 485.343673] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 492.450135] LustreError: 39485:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305179 with bad export cookie 8347396495197643140 [ 492.454496] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 501.329735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.408914] Lustre: Evicted from MGS (at 192.168.202.138@tcp) after server handle changed from 0x73d7e8e2cf98a184 to 0x73d7e8e2cf98a600 [ 509.413331] Lustre: MGC192.168.202.138@tcp: Connection restored to (at 0@lo) [ 509.551786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 509.553686] Lustre: Skipped 9 previous similar messages [ 509.560375] BUG: spinlock bad magic on CPU#1, lctl/43411 [ 509.562204] lock: 0xffff9b6a82f2f100, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 509.565059] CPU: 1 PID: 43411 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 509.569361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 509.571709] Call Trace: [ 509.572276] ? dump_stack+0xbb/0x10e [ 509.574490] ? spin_dump.cold.3+0x32/0x65 [ 509.575867] ? do_raw_spin_lock+0xce/0x190 [ 509.576928] ? _raw_spin_lock+0x21/0x30 [ 509.578036] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 509.579658] ? __seccomp_filter+0x4b/0x680 [ 509.580674] ? full_proxy_write+0x5e/0xa0 [ 509.581550] ? __vfs_write+0x1c/0x60 [ 509.582769] ? vfs_write+0xd8/0x2b0 [ 509.583599] ? ksys_write+0x66/0x120 [ 509.584657] ? __x64_sys_write+0x1e/0x30 [ 509.585798] ? do_syscall_64+0xc1/0x440 [ 509.587316] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 510.993854] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 513.848752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.464881] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 517.914739] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 522.548627] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 524.260977] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 524.310591] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 524.791708] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 528.140483] Lustre: server umount lustre-MDT0000 complete [ 528.141742] Lustre: Skipped 7 previous similar messages [ 528.356797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.360364] Lustre: Skipped 11 previous similar messages [ 529.351214] LustreError: 42554:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305216 with bad export cookie 8347396495197644288 [ 529.353682] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.354471] LustreError: 42554:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 539.103123] Lustre: 45627:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305220/real 1772305220] req@000000004ffb774d x1858396208916032/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772305226 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 539.108890] Lustre: 45627:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 550.240436] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 14:00:37 (1772305237) [ 556.467929] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 560.469237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.043416] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 564.782744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.334155] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 569.318540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 570.795662] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 573.501398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 574.987450] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 583.528440] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 584.709882] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 584.712688] Lustre: Skipped 2 previous similar messages [ 595.936063] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 595.940519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 595.942115] Lustre: Skipped 19 previous similar messages [ 597.459468] LustreError: 47234:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305284 with bad export cookie 8347396495197645051 [ 597.465152] LustreError: 47234:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 597.465204] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 607.199267] Lustre: 50836:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305288/real 1772305288] req@00000000a6860e45 x1858396208938176/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772305294 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 607.206583] Lustre: 50836:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 617.661596] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 14:01:44 (1772305304) [ 618.482585] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 619.128873] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 14:01:46 (1772305306) [ 622.327145] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 624.784108] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 627.805718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 629.719325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 631.917048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 631.943479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.031249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 632.069270] Lustre: lustre-MDT0000: new disk, initializing [ 632.099405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 633.449681] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 637.012403] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 639.073960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 639.095043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 639.098149] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 640.523420] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 645.174114] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 646.256338] Lustre: DEBUG MARKER: create test files [ 656.353616] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 672.884245] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 673.816957] Key type lgssc unregistered [ 673.926279] LNet: 56133:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.928705] LNet: Removed LNI 192.168.202.138@tcp [ 674.228398] Key type .llcrypt unregistered [ 674.229576] Key type ._llcrypt unregistered [ 684.858989] alg: No test for adler32 (adler32-zlib) [ 685.610287] Key type ._llcrypt registered [ 685.611260] Key type .llcrypt registered [ 685.647727] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 686.031884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 686.188119] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 686.262033] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 686.264653] LNet: Accept secure, port 988 [ 687.839159] Key type lgssc registered [ 688.163601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 690.593967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.742730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 693.248229] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 695.654299] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 698.059377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 698.194621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.896956] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 702.671092] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 706.342157] Lustre: lustre-OST0000: deleting orphan objects from 0x0:102 to 0x0:129 [ 706.375814] Lustre: 58970:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 706.381500] Lustre: 58970:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 706.386518] Lustre: 58970:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 706.391594] Lustre: 58970:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 706.396524] Lustre: 58970:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 706.401354] Lustre: 58970:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 711.318335] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 713.128913] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 716.769564] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 716.769939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.778627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 721.135611] Lustre: server umount lustre-OST0000 complete [ 728.543082] Lustre: 59985:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305409/real 1772305409] req@000000001c97a334 x1858396878804224/t0(0) o251->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772305415 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 728.604271] Lustre: server umount lustre-MDT0000 complete [ 733.234590] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 734.203968] Key type lgssc unregistered [ 734.307314] LNet: 60587:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.310133] LNet: Removed LNI 192.168.202.138@tcp [ 734.567069] Key type .llcrypt unregistered [ 734.568415] Key type ._llcrypt unregistered [ 745.140300] alg: No test for adler32 (adler32-zlib) [ 745.891376] Key type ._llcrypt registered [ 745.892462] Key type .llcrypt registered [ 745.934513] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 746.336798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 746.587236] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 746.661093] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 746.663726] LNet: Accept secure, port 988 [ 748.255107] Key type lgssc registered [ 748.786765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 752.096339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.280708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 755.019874] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 757.822936] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 760.589269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 760.787621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 762.959230] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 766.032445] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 768.934085] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 768.962594] Lustre: 63522:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 768.968503] Lustre: 63522:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 768.972912] Lustre: 63522:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 768.978578] Lustre: 63522:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 768.983424] Lustre: 63522:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 768.988549] Lustre: 63522:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 774.563134] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 779.234166] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 779.234836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.247398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 782.757472] Lustre: server umount lustre-OST0000 complete [ 790.495177] Lustre: 64484:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305471/real 1772305471] req@000000004ca302ab x1858396941718784/t0(0) o251->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772305477 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 790.587534] Lustre: server umount lustre-MDT0000 complete [ 795.421650] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 796.355218] Key type lgssc unregistered [ 796.463432] LNet: 65088:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.466800] LNet: Removed LNI 192.168.202.138@tcp [ 796.764615] Key type .llcrypt unregistered [ 796.765928] Key type ._llcrypt unregistered [ 803.596953] alg: No test for adler32 (adler32-zlib) [ 804.348325] Key type ._llcrypt registered [ 804.349327] Key type .llcrypt registered [ 804.384341] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 810.290758] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 810.836166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 811.017696] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 811.096269] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 811.098088] LNet: Accept secure, port 988 [ 812.679116] Key type lgssc registered [ 813.062902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 816.096711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 818.520259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 820.384706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 822.774187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 825.278033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 825.305889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 826.408122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 826.419031] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 826.469727] Lustre: lustre-MDT0000: new disk, initializing [ 826.497316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 826.504685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 828.080630] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 832.186267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 832.212147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.246515] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 832.271695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 832.272347] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 832.275326] Lustre: Skipped 1 previous similar message [ 832.319231] Lustre: lustre-MDT0001: new disk, initializing [ 832.349666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 832.362227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 832.364774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 833.694180] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 837.379216] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 839.200163] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 841.291215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 841.311640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 841.313750] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 841.398107] Lustre: lustre-OST0000: new disk, initializing [ 841.400265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 841.423569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.709907] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 846.445349] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 848.007935] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 848.884633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 848.887766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 849.102169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 850.474284] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 850.546066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 853.985653] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 853.985847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 853.986434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 853.987892] LustreError: Skipped 1 previous similar message [ 853.994610] Lustre: Skipped 1 previous similar message [ 857.301808] Lustre: server umount lustre-OST0000 complete [ 859.105829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.106877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 859.112890] Lustre: Skipped 1 previous similar message [ 859.117085] Lustre: Skipped 1 previous similar message [ 864.224117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 864.226102] Lustre: Skipped 2 previous similar messages [ 864.711478] Lustre: server umount lustre-MDT0000 complete [ 865.961515] LustreError: 68350:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305553 with bad export cookie 2115566458175800942 [ 865.964579] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.967855] LustreError: 68350:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 866.102684] Lustre: server umount lustre-MDT0001 complete [ 868.758809] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 14:05:55 (1772305555) [ 871.991286] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 876.121540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 884.085690] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 887.103628] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 890.468713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 892.704237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 894.667563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 896.797549] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 898.972516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 898.999713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.088238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 899.101427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 899.144764] Lustre: lustre-MDT0000: new disk, initializing [ 899.167884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.173819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 900.641852] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 904.848253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 904.876115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.913147] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 904.933429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 904.935854] Lustre: Skipped 1 previous similar message [ 904.981283] Lustre: lustre-MDT0001: new disk, initializing [ 905.013561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 905.025880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 905.028997] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 906.613119] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 910.302659] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 912.136346] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 914.336286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 914.362311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 914.365089] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 914.483121] Lustre: lustre-OST0000: new disk, initializing [ 914.485122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 914.513563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 915.188361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 915.191697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 915.941121] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 919.700761] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 921.212268] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 921.285080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 922.777599] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 922.864185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 925.667637] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 925.668278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.669103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.669941] LustreError: Skipped 1 previous similar message [ 925.673437] Lustre: Skipped 1 previous similar message [ 929.768690] Lustre: server umount lustre-OST0000 complete [ 935.904645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.904743] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 935.905233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.905237] Lustre: Skipped 1 previous similar message [ 935.907773] Lustre: Skipped 1 previous similar message [ 937.161311] Lustre: server umount lustre-MDT0000 complete [ 938.445220] LustreError: 76647:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305625 with bad export cookie 2115566458175801796 [ 938.448507] LustreError: 76647:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 938.449136] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 941.254684] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 14:07:08 (1772305628) [ 943.373751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.535413] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 943.555390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.963274] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 947.716029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.326854] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 951.969582] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 954.101632] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 956.619309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 958.139674] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 960.602347] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 961.900506] Lustre: 83943:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting a5cf7f76-7ac7-4612-8e44-56e243f814c8 at adminstrative request [ 962.919411] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 962.930878] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 968.161179] Lustre: MGS: Client a5cf7f76-7ac7-4612-8e44-56e243f814c8 (at 0@lo) reconnecting [ 968.166156] Lustre: Evicted from MGS (at 192.168.202.138@tcp) after server handle changed from 0x1d5c007c264ccd0c to 0x1d5c007c264ccf9e [ 968.171606] Lustre: MGC192.168.202.138@tcp: Connection restored to 192.168.202.138@tcp (at 0@lo) [ 994.118788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 994.143994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 994.146734] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 994.192910] Lustre: lustre-OST0001: new disk, initializing [ 994.194738] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 994.223413] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 994.225310] Lustre: Skipped 2 previous similar messages [ 995.556186] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 999.708908] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1000.962645] Lustre: server umount lustre-OST0001 complete [ 1000.963955] Lustre: Skipped 1 previous similar message [ 1002.992080] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.994739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1003.007312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1003.010605] Lustre: Skipped 2 previous similar messages [ 1008.100459] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.108920] LustreError: Skipped 3 previous similar messages [ 1013.218106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.227124] Lustre: Skipped 1 previous similar message [ 1017.097130] LustreError: 84334:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305704 with bad export cookie 2115566458175803294 [ 1017.100018] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1017.100608] LustreError: 84334:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1017.220368] Lustre: server umount lustre-MDT0001 complete [ 1017.222063] Lustre: Skipped 2 previous similar messages [ 1020.220728] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 14:08:27 (1772305707) [ 1020.710879] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1021.221422] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 14:08:28 (1772305708) [ 1021.728717] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1022.332214] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 14:08:29 (1772305709) [ 1023.004289] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1023.650816] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 14:08:30 (1772305710) [ 1024.174706] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1026.387884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1026.727940] loop: module loaded [ 1026.732927] loop0: detected capacity change from 0 to 4294967296 [ 1026.751524] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1027.259270] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1028.359172] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1029.632581] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1032.621460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1034.289624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1036.045537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1037.820732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1040.880127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1040.919970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1040.925751] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1040.934044] LustreError: 88379:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1040.938930] LustreError: 88379:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1040.943964] LustreError: 88379:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1040.949913] LustreError: 88379:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1049.091303] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 1052.169952] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1056.691078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1059.577129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1062.565260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1065.249554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1068.436828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1068.480134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.616417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1068.630085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1068.675289] Lustre: lustre-MDT0000: new disk, initializing [ 1068.704537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.712512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1070.501776] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1074.916891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1074.945142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1075.099964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1076.822868] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1081.322344] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1083.607679] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1086.132561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1086.171132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1086.180142] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1087.863369] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1092.273132] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1094.137963] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1094.150806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1095.307878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1096.807754] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1096.889520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1099.233452] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1099.235106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.236284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1099.239145] Lustre: Skipped 2 previous similar messages [ 1099.242008] Lustre: Skipped 7 previous similar messages [ 1103.735406] Lustre: server umount lustre-OST0000 complete [ 1105.888370] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1105.892491] LustreError: Skipped 1 previous similar message [ 1112.885316] LustreError: 91230:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305799 with bad export cookie 2115566458175803833 [ 1112.889066] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1112.890926] LustreError: 91230:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1116.565789] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 14:10:03 (1772305803) [ 1119.480678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.686741] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1121.419704] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1124.835199] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.519320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1127.610810] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1130.715544] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1132.746291] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1135.519539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1137.289442] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1140.135984] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1142.705654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1142.732279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1142.734757] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1142.779309] Lustre: lustre-OST0001: new disk, initializing [ 1142.781060] Lustre: Skipped 2 previous similar messages [ 1142.782649] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1142.784414] Lustre: Skipped 3 previous similar messages [ 1144.216443] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1148.665474] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1149.943909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1149.947695] Lustre: Skipped 2 previous similar messages [ 1149.949848] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1159.083378] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1164.952558] Lustre: Failing over lustre-MDT0000 [ 1165.281805] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.283020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.286367] Lustre: Skipped 4 previous similar messages [ 1165.288562] Lustre: Skipped 4 previous similar messages [ 1165.415172] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.419278] LustreError: Skipped 1 previous similar message [ 1177.568120] Lustre: 66587:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305857/real 1772305857] req@00000000b5a32cc5 x1858397009963904/t0(0) o400->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772305864 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1177.576257] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1182.306637] LDISKFS-fs (dm-0): recovery complete [ 1182.309587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1182.622074] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.626787] LustreError: Skipped 23 previous similar messages [ 1183.720289] Lustre: Evicted from MGS (at 192.168.202.138@tcp) after server handle changed from 0x1d5c007c264cd501 to 0x1d5c007c264d62e4 [ 1183.727326] Lustre: MGC192.168.202.138@tcp: Connection restored to (at 0@lo) [ 1183.862081] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1185.621821] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1187.933431] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1187.933480] Lustre: 95788:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1189.352229] Lustre: 95792:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1189.357500] Lustre: 95792:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 1 previous similar message [ 1189.367349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1189.400770] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1189.719148] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1190.042259] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1190.047896] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 1190.367133] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1190.371983] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 1190.377773] Lustre: 100945:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1190.384758] Lustre: 100945:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 7 previous similar messages [ 1191.330458] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1191.336291] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 1191.655221] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1191.659931] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 1192.610484] Lustre: 100945:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1192.618748] Lustre: 100945:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 6 previous similar messages [ 1193.560210] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1193.564876] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 1193.879136] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1193.883037] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 1196.705166] Lustre: 100945:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1196.714145] Lustre: 100945:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages [ 1197.658037] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1197.663455] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 1197.983117] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1197.987844] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 1204.864235] Lustre: 100945:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1204.870169] Lustre: 100945:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 25 previous similar messages [ 1205.809124] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1205.812205] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 1206.127123] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1206.130829] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 1221.120464] Lustre: 100945:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1221.129531] Lustre: 100945:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 51 previous similar messages [ 1222.082732] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1222.088918] LustreError: 100945:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 51 previous similar messages [ 1222.407204] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1222.412260] LustreError: 100945:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 51 previous similar messages [ 1247.087133] Lustre: 100945:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1247.091404] Lustre: 100945:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1247.095346] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1247.097647] LustreError: 100945:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 1247.103620] Lustre: 100945:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@00000000ca4f0335 x1858397004657024/t0(8589934781) o101->4572faed-0fac-4455-8c9c-a1c1baa2f9c2@192.168.202.38@tcp:64/0 lens 592/0 e 0 to 0 dl 1772305939 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 1247.116081] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1247.119489] Lustre: Skipped 2 previous similar messages [ 1247.120565] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1247.143620] Lustre: lustre-OST0000: deleting orphan objects from 0x0:80 to 0x0:577 [ 1247.143628] Lustre: lustre-OST0001: deleting orphan objects from 0x0:48 to 0x0:545 [ 1250.785721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.796479] Lustre: Skipped 2 previous similar messages [ 1256.631377] Lustre: server umount lustre-OST0000 complete [ 1256.634190] Lustre: Skipped 3 previous similar messages [ 1261.025621] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.030698] LustreError: Skipped 8 previous similar messages [ 1273.224020] LustreError: 95773:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305960 with bad export cookie 2115566458175840996 [ 1273.226803] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.226949] LustreError: 95773:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1276.079308] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 14:12:43 (1772305963) [ 1278.562950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1278.565325] Lustre: *** cfs_fail_loc=197, val=0*** [ 1295.007517] LustreError: 102966:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 1: rc = -5 [ 1306.271343] LustreError: 102966:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 4: rc = -110 [ 1312.421892] Lustre: *** cfs_fail_loc=197, val=0*** [ 1312.423091] Lustre: Skipped 65 previous similar messages [ 1335.967268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1335.969907] Lustre: Skipped 7 previous similar messages [ 1337.278558] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1339.655981] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1343.098781] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 14:13:50 (1772306030) [ 1356.874289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.016434] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.406870] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1362.220492] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1364.253761] LustreError: 104624:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1364.256629] LustreError: 104624:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1364.259248] LustreError: 104624:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1370.591421] Lustre: 105313:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306051/real 1772306051] req@000000006b415590 x1858397009991872/t0(0) o251->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772306057 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1375.626436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1381.135836] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 1384.057463] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1387.202653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1389.204414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1391.075078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1393.045542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1395.226770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1395.252583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.333151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1395.334892] Lustre: Skipped 1 previous similar message [ 1395.342354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1395.377478] Lustre: lustre-MDT0000: new disk, initializing [ 1395.405322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1396.680437] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1401.014843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1401.037779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1401.160736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1402.472920] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1406.357660] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1408.257235] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1410.336662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1410.358213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1410.360542] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1411.957239] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1415.693062] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1417.009413] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1419.126558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1420.529739] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1420.615260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1423.330946] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1423.331264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.332536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1423.332543] Lustre: Skipped 12 previous similar messages [ 1423.335048] LustreError: Skipped 1 previous similar message [ 1423.347582] Lustre: Skipped 5 previous similar messages [ 1427.345357] Lustre: server umount lustre-OST0000 complete [ 1427.346726] Lustre: Skipped 5 previous similar messages [ 1435.868103] LustreError: 108931:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306122 with bad export cookie 2115566458175851664 [ 1435.871145] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1435.872797] LustreError: 108931:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1438.703545] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 14:15:25 (1772306125) [ 1442.924449] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1443.754963] Key type lgssc unregistered [ 1443.869292] LNet: 113628:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.871776] LNet: Removed LNI 192.168.202.138@tcp [ 1444.138274] Key type .llcrypt unregistered [ 1444.139428] Key type ._llcrypt unregistered [ 1449.775027] alg: No test for adler32 (adler32-zlib) [ 1450.526377] Key type ._llcrypt registered [ 1450.527332] Key type .llcrypt registered [ 1450.563855] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 1456.092146] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1456.491573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1456.663071] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1456.729415] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1456.731114] LNet: Accept secure, port 988 [ 1458.311085] Key type lgssc registered [ 1458.608129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1462.007728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1464.700645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1467.347947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1470.157636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1472.191865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1474.547306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1474.583576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.718415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1475.733347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1475.794527] Lustre: lustre-MDT0000: new disk, initializing [ 1475.826154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.834328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1477.910509] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1482.876219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1482.902040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1482.905550] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1483.016572] Lustre: lustre-OST0000: new disk, initializing [ 1483.018946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1483.021355] Lustre: Skipped 1 previous similar message [ 1483.046550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.824211] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1489.129442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1489.157372] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1489.160440] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1489.214794] Lustre: lustre-OST0001: new disk, initializing [ 1489.217652] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1489.255557] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1490.934567] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1495.120137] Lustre: server umount lustre-MDT0000 complete [ 1497.566657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1499.092113] LustreError: 117108:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306186 with bad export cookie 580602629384155699 [ 1499.095514] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1505.247213] Lustre: 119945:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306186/real 1772306186] req@00000000c2eafb43 x1858397686211200/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772306192 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1505.283478] Lustre: server umount lustre-OST0000 complete [ 1506.759956] Lustre: 115126:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306186/real 1772306186] req@0000000047709b7c x1858397686211392/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772306193 ref 1 fl Rpc:EXNQr/0/ffffffff rc -5/-1 job:'' [ 1512.927178] Lustre: 120147:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306193/real 1772306193] req@00000000fa4eb181 x1858397686211648/t0(0) o39->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772306199 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1513.043315] Lustre: server umount lustre-OST0001 complete [ 1515.976390] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 14:16:42 (1772306202) [ 1518.918223] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1522.106469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1522.436236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1524.429981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1524.855468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1527.499898] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 14:16:54 (1772306214) [ 1532.091304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1532.875711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1533.498488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1534.175407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1534.550438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1535.365395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1541.741028] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 1544.452667] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1547.720024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1549.796821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1551.646935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1554.009190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1556.388545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1556.410345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1556.490050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1556.501728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1556.542950] Lustre: lustre-MDT0000: new disk, initializing [ 1556.563981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.570156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1557.879281] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1562.523346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1562.559413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.596903] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1562.625418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1562.627431] Lustre: Skipped 1 previous similar message [ 1562.633203] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1562.668772] Lustre: lustre-MDT0001: new disk, initializing [ 1562.699807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1562.713479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1562.717463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1564.210851] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1568.880368] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1571.281950] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1574.357280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1574.393251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1574.399650] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1574.531072] Lustre: lustre-OST0000: new disk, initializing [ 1574.533307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1574.566964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.221268] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1577.040166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1577.046492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1580.457130] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1581.828260] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1581.895662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1583.231888] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1583.302442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1587.682575] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1587.682658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1587.686570] LustreError: Skipped 1 previous similar message [ 1587.692845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1587.696100] Lustre: Skipped 1 previous similar message [ 1590.037148] Lustre: server umount lustre-OST0000 complete [ 1592.800613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.801188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1592.806367] Lustre: Skipped 2 previous similar messages [ 1592.811287] Lustre: Skipped 1 previous similar message [ 1597.386185] Lustre: server umount lustre-MDT0000 complete [ 1597.919989] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.925454] LustreError: Skipped 1 previous similar message [ 1598.632951] LustreError: 125629:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306285 with bad export cookie 580602629384156847 [ 1598.635972] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1598.636225] LustreError: 125629:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1601.394704] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 14:18:08 (1772306288) [ 1603.415681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1603.560346] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1603.580788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1604.872717] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1607.558219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.088189] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1611.513396] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1613.339834] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1615.565932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1617.251188] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1619.908608] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1625.707401] Lustre: *** cfs_fail_loc=159, val=0*** [ 1625.710289] Lustre: Skipped 2 previous similar messages [ 1632.872518] Lustre: lustre-MDT0000: Client d52bbc0d-3e8f-40fb-a7f8-436a37569430 (at 192.168.202.38@tcp) reconnecting [ 1633.200077] Lustre: *** cfs_fail_loc=159, val=0*** [ 1633.201389] Lustre: Skipped 3 previous similar messages [ 1639.528105] Lustre: lustre-MDT0000: Client d52bbc0d-3e8f-40fb-a7f8-436a37569430 (at 192.168.202.38@tcp) reconnecting [ 1643.491446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.493527] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1643.493724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1643.495775] Lustre: Skipped 1 previous similar message [ 1646.651911] Lustre: server umount lustre-OST0000 complete [ 1646.654163] Lustre: Skipped 1 previous similar message [ 1648.609702] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1648.610058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.621156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1648.625066] Lustre: Skipped 2 previous similar messages [ 1655.774850] LustreError: 130167:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306342 with bad export cookie 580602629384157701 [ 1655.778462] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1655.782275] LustreError: 130167:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1660.999935] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1661.960442] Key type lgssc unregistered [ 1662.076361] LNet: 134039:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.080190] LNet: Removed LNI 192.168.202.138@tcp [ 1662.485929] Key type .llcrypt unregistered [ 1662.488139] Key type ._llcrypt unregistered [ 1672.338211] alg: No test for adler32 (adler32-zlib) [ 1673.089407] Key type ._llcrypt registered [ 1673.090463] Key type .llcrypt registered [ 1673.130168] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1673.542392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1673.685205] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1673.753252] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1673.755042] LNet: Accept secure, port 988 [ 1675.327078] Key type lgssc registered [ 1675.645349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1678.330494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.478052] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1679.493924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1680.920361] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1684.026387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.150460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1685.495033] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1687.879691] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1689.641621] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1691.773461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1691.886264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1693.303714] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1694.948662] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 1695.713917] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1705.276682] Lustre: *** cfs_fail_loc=159, val=0*** [ 1711.718063] Lustre: lustre-MDT0000: Client da217f00-677e-4307-93e3-0104c7e08990 (at 192.168.202.38@tcp) reconnecting [ 1712.300835] Lustre: *** cfs_fail_loc=159, val=0*** [ 1719.398594] Lustre: lustre-MDT0001: Client da217f00-677e-4307-93e3-0104c7e08990 (at 192.168.202.38@tcp) reconnecting [ 1719.772718] Lustre: *** cfs_fail_loc=159, val=0*** [ 1719.775778] Lustre: Skipped 5 previous similar messages [ 1727.098530] Lustre: lustre-MDT0001: Client da217f00-677e-4307-93e3-0104c7e08990 (at 192.168.202.38@tcp) reconnecting [ 1728.935126] Lustre: *** cfs_fail_loc=159, val=0*** [ 1728.937543] Lustre: Skipped 3 previous similar messages [ 1736.296836] Lustre: lustre-MDT0000: Client 278e520c-522f-47f6-b2a5-3ad8d722ccde (at 192.168.202.38@tcp) reconnecting [ 1736.652046] Lustre: *** cfs_fail_loc=159, val=0*** [ 1736.653594] Lustre: Skipped 13 previous similar messages [ 1743.980617] Lustre: lustre-MDT0000: Client 278e520c-522f-47f6-b2a5-3ad8d722ccde (at 192.168.202.38@tcp) reconnecting [ 1746.399952] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1746.402138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.405814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1749.986611] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 1749.990245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1749.996180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1751.858852] Lustre: server umount lustre-OST0000 complete [ 1755.617428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.618850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1755.624503] Lustre: Skipped 1 previous similar message [ 1755.630814] Lustre: Skipped 3 previous similar messages [ 1759.704383] Lustre: server umount lustre-MDT0000 complete [ 1760.736697] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1760.743848] LustreError: Skipped 1 previous similar message [ 1761.208390] LustreError: 135484:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306448 with bad export cookie 5482294090160704620 [ 1761.211938] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1761.215580] LustreError: 135484:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1761.371248] Lustre: server umount lustre-MDT0001 complete [ 1766.560842] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1767.692075] Key type lgssc unregistered [ 1767.809377] LNet: 139886:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.814239] LNet: Removed LNI 192.168.202.138@tcp [ 1768.145957] Key type .llcrypt unregistered [ 1768.147605] Key type ._llcrypt unregistered [ 1778.458341] alg: No test for adler32 (adler32-zlib) [ 1779.210506] Key type ._llcrypt registered [ 1779.212454] Key type .llcrypt registered [ 1779.256566] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1779.632463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1779.953470] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1780.033708] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1780.035230] LNet: Accept secure, port 988 [ 1781.631128] Key type lgssc registered [ 1782.043881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1784.775615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.929487] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.942657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1787.510708] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1789.986511] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.025991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.174280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1792.794168] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1795.711256] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1797.613306] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1800.209947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1800.308325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.322525] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:129 [ 1801.846186] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1804.545091] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1806.444722] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 1813.030263] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1814.187407] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1814.190194] Lustre: Skipped 1 previous similar message [ 1817.061454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.073393] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1817.073793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.083557] Lustre: Skipped 1 previous similar message [ 1821.974719] Lustre: server umount lustre-OST0000 complete [ 1827.297708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.298254] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1827.298739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1827.305714] Lustre: Skipped 2 previous similar messages [ 1827.311119] LustreError: Skipped 1 previous similar message [ 1829.275976] Lustre: server umount lustre-MDT0000 complete [ 1830.922243] LustreError: 144011:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306517 with bad export cookie 2822945644125648260 [ 1830.925080] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.925640] LustreError: 144011:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1831.046462] Lustre: server umount lustre-MDT0001 complete [ 1835.987437] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1836.937691] Key type lgssc unregistered [ 1837.032341] LNet: 145400:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.034870] LNet: Removed LNI 192.168.202.138@tcp [ 1837.335639] Key type .llcrypt unregistered [ 1837.336770] Key type ._llcrypt unregistered [ 1846.788229] alg: No test for adler32 (adler32-zlib) [ 1847.539464] Key type ._llcrypt registered [ 1847.540588] Key type .llcrypt registered [ 1847.593587] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1847.949295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1848.255564] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1848.326021] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1848.328305] LNet: Accept secure, port 988 [ 1849.911099] Key type lgssc registered [ 1850.318184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1853.517961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.681556] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1854.698600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.226608] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1859.418545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.557788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1861.003271] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1863.379992] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1865.186129] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1867.317519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1867.437786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.829489] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1870.506646] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:161 [ 1871.485808] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1874.600723] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 1878.598264] Lustre: *** cfs_fail_loc=159, val=0*** [ 1885.799626] Lustre: lustre-MDT0000: Client 24dea4ed-1e00-4244-b8ce-75da276d2a2d (at 192.168.202.38@tcp) reconnecting [ 1890.274972] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1890.276235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.278985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1890.284884] Lustre: Skipped 1 previous similar message [ 1892.977721] Lustre: server umount lustre-OST0000 complete [ 1895.394519] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1895.395421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.396105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1895.396115] Lustre: Skipped 1 previous similar message [ 1895.416983] Lustre: Skipped 1 previous similar message [ 1900.428646] Lustre: server umount lustre-MDT0000 complete [ 1900.512086] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.516534] LustreError: Skipped 1 previous similar message [ 1901.656428] LustreError: 146905:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306588 with bad export cookie 4373585616169935010 [ 1901.659589] LustreError: 146905:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1901.660620] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1901.796903] Lustre: server umount lustre-MDT0001 complete [ 1906.659425] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1907.680791] Key type lgssc unregistered [ 1907.793383] LNet: 150641:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.796098] LNet: Removed LNI 192.168.202.138@tcp [ 1908.059255] Key type .llcrypt unregistered [ 1908.060449] Key type ._llcrypt unregistered [ 1917.038508] alg: No test for adler32 (adler32-zlib) [ 1917.790292] Key type ._llcrypt registered [ 1917.791525] Key type .llcrypt registered [ 1917.825230] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1918.097456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1918.325297] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1918.396996] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1918.399154] LNet: Accept secure, port 988 [ 1919.975084] Key type lgssc registered [ 1920.351402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1923.790046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1924.939046] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1924.961405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.495200] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1930.070828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.238034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1931.802204] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1934.700974] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1937.037817] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1939.742874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1939.919806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.966782] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 1941.680873] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 1944.646306] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1945.061030] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 1958.579766] Lustre: lustre-OST0000: evicting ac0653c1-fc28-45b3-86ba-da85855d6e23 (at 192.168.202.38@tcp) by administrative request [ 1958.582191] Lustre: Skipped 1 previous similar message [ 1986.017186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.018594] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1986.021955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1986.026375] Lustre: Skipped 1 previous similar message [ 1991.136568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1991.138278] Lustre: Skipped 1 previous similar message [ 1996.256324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1996.259010] Lustre: Skipped 1 previous similar message [ 1997.279110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1997.337093] Lustre: server umount lustre-OST0000 complete [ 2001.376948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2001.378128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2001.386681] Lustre: Skipped 2 previous similar messages [ 2004.630489] Lustre: server umount lustre-MDT0000 complete [ 2006.220867] LustreError: 152082:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306693 with bad export cookie 13005066138786169391 [ 2006.223437] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2006.223780] LustreError: 152082:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2006.341612] Lustre: server umount lustre-MDT0001 complete [ 2011.366931] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2012.569230] Key type lgssc unregistered [ 2012.705565] LNet: 156297:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2012.709833] LNet: Removed LNI 192.168.202.138@tcp [ 2013.040100] Key type .llcrypt unregistered [ 2013.041326] Key type ._llcrypt unregistered [ 2036.649528] alg: No test for adler32 (adler32-zlib) [ 2037.401313] Key type ._llcrypt registered [ 2037.402408] Key type .llcrypt registered [ 2037.438468] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2037.815412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2038.064434] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 2038.146639] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2038.148246] LNet: Accept secure, port 988 [ 2039.727075] Key type lgssc registered [ 2040.030569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2042.455298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.584524] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.600146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.019247] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2047.724289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.847453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2049.181617] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2051.512036] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2053.328816] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2055.470310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2055.581649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.909999] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2058.660503] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 2059.355639] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2062.756632] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 2067.937192] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2067.937251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.938170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2067.947689] Lustre: Skipped 1 previous similar message [ 2073.057356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2073.060359] Lustre: Skipped 2 previous similar messages [ 2073.235262] Lustre: server umount lustre-OST0000 complete [ 2078.176391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.176946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2078.179402] Lustre: Skipped 1 previous similar message [ 2080.647084] Lustre: server umount lustre-MDT0000 complete [ 2081.876139] LustreError: 161484:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306768 with bad export cookie 3065873966024843910 [ 2081.878545] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2081.882337] LustreError: 161484:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2082.004110] Lustre: server umount lustre-MDT0001 complete [ 2086.602068] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2087.702912] Key type lgssc unregistered [ 2087.819455] LNet: 162487:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.822447] LNet: Removed LNI 192.168.202.138@tcp [ 2088.122858] Key type .llcrypt unregistered [ 2088.123865] Key type ._llcrypt unregistered [ 2096.650833] alg: No test for adler32 (adler32-zlib) [ 2097.402483] Key type ._llcrypt registered [ 2097.404064] Key type .llcrypt registered [ 2097.443216] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2097.757979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2097.996762] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 2098.071506] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2098.073340] LNet: Accept secure, port 988 [ 2099.647094] Key type lgssc registered [ 2099.975023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2102.203860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2104.483333] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 14:26:31 (1772306791) [ 2109.416157] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 2111.996946] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2115.061501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2117.064737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2118.882779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2120.783645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2122.864781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2122.887983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.967407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2123.977665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2124.012919] Lustre: lustre-MDT0000: new disk, initializing [ 2124.034583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2124.040850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2125.354071] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2129.235881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2129.261450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2129.291041] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2129.304821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2129.306518] Lustre: Skipped 1 previous similar message [ 2129.340211] Lustre: lustre-MDT0001: new disk, initializing [ 2129.366519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2129.376306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2129.378757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2130.652411] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2134.260234] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2136.007354] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2138.077144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2138.098271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2138.101074] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2138.180449] Lustre: lustre-OST0000: new disk, initializing [ 2138.182204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2138.210137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.497953] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2139.828246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2139.831214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2143.182593] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2148.311597] Lustre: server umount lustre-OST0000 complete [ 2149.855754] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2149.858021] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.861477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2150.368020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2150.368548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.975114] Lustre: 170078:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306836/real 1772306836] req@00000000cda8cb8a x1858398359401280/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772306842 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2155.048185] Lustre: server umount lustre-MDT0000 complete [ 2155.487974] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.491506] LustreError: Skipped 1 previous similar message [ 2156.236907] LustreError: 166877:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306843 with bad export cookie 18188262679704360600 [ 2156.239450] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2156.239850] LustreError: 166877:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2162.655096] Lustre: 170279:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306843/real 1772306843] req@0000000065d99c9e x1858398359402560/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772306849 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2162.779496] Lustre: server umount lustre-MDT0001 complete [ 2167.224376] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2168.136720] Key type lgssc unregistered [ 2168.240256] LNet: 170880:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.242485] LNet: Removed LNI 192.168.202.138@tcp [ 2168.522654] Key type .llcrypt unregistered [ 2168.524566] Key type ._llcrypt unregistered [ 2176.896603] alg: No test for adler32 (adler32-zlib) [ 2177.647323] Key type ._llcrypt registered [ 2177.648328] Key type .llcrypt registered [ 2177.680843] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2177.971557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2178.195149] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 2178.262544] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2178.264102] LNet: Accept secure, port 988 [ 2179.839133] Key type lgssc registered [ 2180.130635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2182.488776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.617706] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.630636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.930755] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2187.572193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.686962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2189.031508] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2191.302521] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2193.020545] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2195.056288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2195.154533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.455684] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2198.245153] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2198.766939] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2205.606350] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2205.611887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.616044] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2205.618343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2205.621806] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.138@tcp (at 0@lo) [ 2205.622574] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2205.929898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2205.933884] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 2206.236501] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2206.239100] Lustre: Skipped 1 previous similar message [ 2206.243794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.247154] Lustre: Skipped 1 previous similar message [ 2206.248516] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2206.250218] Lustre: Skipped 1 previous similar message [ 2206.251319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2206.254279] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.138@tcp (at 0@lo) [ 2206.254435] Lustre: lustre-OST0000: deleting orphan objects from 0x0:186 to 0x0:225 [ 2206.256081] Lustre: Skipped 1 previous similar message [ 2206.528294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2206.531522] Lustre: lustre-OST0000: deleting orphan objects from 0x0:186 to 0x0:257 [ 2206.807197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2206.810702] Lustre: lustre-OST0000: deleting orphan objects from 0x0:186 to 0x0:289 [ 2207.088340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2207.092992] Lustre: lustre-OST0000: deleting orphan objects from 0x0:186 to 0x0:321 [ 2207.367214] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2207.368960] Lustre: Skipped 3 previous similar messages [ 2207.372716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2207.375768] Lustre: Skipped 3 previous similar messages [ 2207.377056] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2207.378982] Lustre: Skipped 3 previous similar messages [ 2207.380256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2207.383973] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.138@tcp (at 0@lo) [ 2207.384228] Lustre: lustre-OST0000: deleting orphan objects from 0x0:186 to 0x0:353 [ 2207.386214] Lustre: Skipped 3 previous similar messages [ 2207.698372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2207.702663] Lustre: lustre-OST0000: deleting orphan objects from 0x0:186 to 0x0:385 [ 2207.992125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2207.995958] Lustre: lustre-OST0000: deleting orphan objects from 0x0:186 to 0x0:417 [ 2208.275862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2208.280181] Lustre: lustre-OST0000: deleting orphan objects from 0x0:186 to 0x0:449 [ 2210.787141] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2210.789325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2210.792122] Lustre: Skipped 3 previous similar messages [ 2210.793521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2212.561743] Lustre: server umount lustre-OST0000 complete [ 2217.952325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.953063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2217.955800] Lustre: Skipped 1 previous similar message [ 2218.975139] Lustre: 171653:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306899/real 1772306899] req@00000000983ba2f3 x1858398443287232/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772306906 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2218.976694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2218.984740] Lustre: 171653:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2218.986301] Lustre: Skipped 2 previous similar messages [ 2224.095949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2224.097850] Lustre: Skipped 1 previous similar message [ 2225.119114] Lustre: 171656:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306905/real 1772306905] req@0000000045b53466 x1858398443288064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772306912 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2228.191123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2228.229288] Lustre: server umount lustre-MDT0000 complete [ 2229.412894] LustreError: 172322:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306916 with bad export cookie 9213863341027807193 [ 2229.415382] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2229.415773] LustreError: 172322:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2229.527197] Lustre: server umount lustre-MDT0001 complete [ 2233.919923] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2234.876099] Key type lgssc unregistered [ 2234.974353] LNet: 176795:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.977135] LNet: Removed LNI 192.168.202.138@tcp [ 2235.250148] Key type .llcrypt unregistered [ 2235.251889] Key type ._llcrypt unregistered [ 2244.194085] alg: No test for adler32 (adler32-zlib) [ 2244.946291] Key type ._llcrypt registered [ 2244.947805] Key type .llcrypt registered [ 2244.989225] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2245.410544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2245.563134] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 2245.633116] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2245.635469] LNet: Accept secure, port 988 [ 2247.215162] Key type lgssc registered [ 2247.744320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2250.494540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.629255] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.642794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.087887] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2255.892318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.019409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2257.499475] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2260.049633] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2262.251710] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2265.373561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2265.540669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2267.284633] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2267.624417] Lustre: lustre-OST0000: deleting orphan objects from 0x0:186 to 0x0:481 [ 2269.805807] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2276.964497] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2277.851690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.859114] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2277.863621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2277.868180] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.138@tcp (at 0@lo) [ 2277.869075] LustreError: 178288:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2281.576780] Lustre: lustre-OST0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 2281.956647] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2281.962065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.866879] Lustre: server umount lustre-OST0000 complete [ 2297.951061] LustreError: 178288:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 2297.953506] LustreError: 178288:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2313.924289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2314.079485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2315.109538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2315.114678] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2315.114836] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2315.116903] Lustre: lustre-OST0000: deleting orphan objects from 0x0:186 to 0x0:513 [ 2315.127959] Lustre: Skipped 1 previous similar message [ 2315.128160] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:66 to 0x280000400:97 [ 2315.474714] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2318.670943] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2319.460878] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2321.656037] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9dfc45266000.ost_server_uuid 40 [ 2322.151709] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9dfc45266000.ost_server_uuid in FULL state after 0 sec [ 2325.473510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.473880] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2325.476714] Lustre: Skipped 1 previous similar message [ 2325.481931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2325.483618] Lustre: Skipped 3 previous similar messages [ 2329.173252] Lustre: server umount lustre-OST0000 complete [ 2330.592814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2330.593655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2330.599711] Lustre: Skipped 2 previous similar messages [ 2330.605729] Lustre: Skipped 1 previous similar message [ 2335.712818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2335.717154] Lustre: Skipped 1 previous similar message [ 2336.580962] Lustre: server umount lustre-MDT0000 complete [ 2337.740838] LustreError: 178249:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307024 with bad export cookie 4594574407287084455 [ 2337.743292] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2337.745379] LustreError: 178249:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2342.359084] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2343.340851] Key type lgssc unregistered [ 2343.449460] LNet: 183559:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.451918] LNet: Removed LNI 192.168.202.138@tcp [ 2343.723786] Key type .llcrypt unregistered [ 2343.724866] Key type ._llcrypt unregistered [ 2354.781215] alg: No test for adler32 (adler32-zlib) [ 2355.532450] Key type ._llcrypt registered [ 2355.533548] Key type .llcrypt registered [ 2355.573724] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2356.102561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2356.317126] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 2356.401063] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2356.403930] LNet: Accept secure, port 988 [ 2357.983138] Key type lgssc registered [ 2358.413454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2361.480803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2362.619073] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.635687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.058129] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2367.320673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.436646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2369.028184] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2371.937763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2372.051594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2373.481602] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2376.319425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2376.341330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2376.343763] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2376.391362] Lustre: lustre-OST0001: new disk, initializing [ 2376.393217] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2376.420731] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2377.775940] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2378.474453] Lustre: lustre-OST0000: deleting orphan objects from 0x0:515 to 0x0:545 [ 2380.792657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2380.795312] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2383.075092] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2389.643617] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2403.295925] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2403.299095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2403.304507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2406.369057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2406.370362] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.375409] Lustre: Skipped 2 previous similar messages [ 2407.816983] Lustre: server umount lustre-MDT0000 complete [ 2409.022395] LustreError: 185314:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307096 with bad export cookie 2912767780256385214 [ 2409.025621] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.026371] LustreError: 185314:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2409.162425] Lustre: server umount lustre-MDT0001 complete [ 2416.607115] Lustre: 189116:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772307097/real 1772307097] req@000000006ecd31fa x1858398629959232/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772307103 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2416.629249] Lustre: server umount lustre-OST0000 complete [ 2423.263109] Lustre: 189317:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772307104/real 1772307104] req@00000000920e52cf x1858398629959744/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772307110 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2423.341328] Lustre: server umount lustre-OST0001 complete [ 2424.355844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2424.384355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2424.728539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2424.747294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2425.095734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2425.118513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2425.469345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2425.502415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2429.664974] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2432.742960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.890827] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2432.909822] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.197783] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2436.874893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.519815] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2441.241619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2441.348547] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.351180] Lustre: Skipped 1 previous similar message [ 2442.403313] LustreError: 137-5: mylustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.665287] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2445.469447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.806956] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2448.295396] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:548 to 0x0:577 [ 2454.826228] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2455.952514] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2467.807851] LustreError: 11-0: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2467.810248] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.813805] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2467.815690] Lustre: Skipped 3 previous similar messages [ 2471.112938] Lustre: server umount mylustre-MDT0000 complete [ 2472.221964] LustreError: 191577:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307159 with bad export cookie 2912767780256387895 [ 2472.224298] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2472.225114] LustreError: 191577:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2479.583234] Lustre: 194475:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772307160/real 1772307160] req@0000000091064fec x1858398629983744/t0(0) o39->mylustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772307166 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2479.606410] Lustre: server umount mylustre-OST0000 complete [ 2479.607718] Lustre: Skipped 1 previous similar message [ 2486.239137] Lustre: 194677:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772307167/real 1772307167] req@0000000026e4fd8a x1858398629984192/t0(0) o39->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772307173 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2487.259689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2487.299288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2487.643400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2487.679950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2488.013968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2488.038627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2488.363663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2488.393258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2492.666139] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2495.677706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.813428] LustreError: 137-5: tfs-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2495.833093] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.834855] Lustre: Skipped 1 previous similar message [ 2497.129665] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2499.882655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.436970] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2504.199216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2505.317779] LustreError: 137-5: tfs-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2505.719266] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2508.453159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2509.887994] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2511.783041] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 2511.783327] Lustre: tfs-OST0000: deleting orphan objects from 0x0:548 to 0x0:609 [ 2518.430491] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2519.598280] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2527.200967] Lustre: tfs-MDT0000-lwp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.202592] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.204096] Lustre: Skipped 5 previous similar messages [ 2527.207076] Lustre: Skipped 7 previous similar messages [ 2532.320120] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 2532.491381] Lustre: server umount tfs-MDT0000 complete [ 2532.492895] Lustre: Skipped 1 previous similar message [ 2533.645082] LustreError: 196115:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307220 with bad export cookie 2912767780256389505 [ 2533.648786] LustreError: 196115:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2533.649382] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2541.023099] Lustre: 199776:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772307222/real 1772307222] req@000000002ee55af4 x1858398630004096/t0(0) o39->tfs-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772307228 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2549.709986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2549.734902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2550.059732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2550.087193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2550.417822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2550.444886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2550.761863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2550.777036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2555.206379] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2558.210986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.348522] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.367848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2558.369737] Lustre: Skipped 3 previous similar messages [ 2559.750834] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2562.481424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.903201] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2567.067268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2568.649445] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2571.719660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2573.035944] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2574.247659] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 2574.247701] Lustre: lustre-OST0000: deleting orphan objects from 0x0:548 to 0x0:641 [ 2581.384713] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2582.524737] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2589.431907] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 14:34:36 (1772307276) [ 2603.488314] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2603.491644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2603.496519] Lustre: Skipped 1 previous similar message [ 2603.498659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2603.501102] Lustre: Skipped 3 previous similar messages [ 2607.647526] Lustre: server umount lustre-MDT0000 complete [ 2607.649582] Lustre: Skipped 3 previous similar messages [ 2609.421565] LustreError: 201419:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307296 with bad export cookie 2912767780256391136 [ 2609.428114] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2609.428628] LustreError: 201419:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2610.144577] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.152468] LustreError: Skipped 2 previous similar messages [ 2633.173637] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 2636.974407] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2641.000991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2644.106995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2646.360111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2648.760574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2651.377858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2651.413571] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 2651.413571] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2651.413571] [ 2651.425769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2651.559691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2651.577598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2651.633491] Lustre: lustre-MDT0000: new disk, initializing [ 2651.657466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.660229] Lustre: Skipped 3 previous similar messages [ 2651.667858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2653.249202] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2658.198122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2658.230966] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 2658.230966] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2658.230966] [ 2658.239036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2658.301955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2658.306547] Lustre: Skipped 1 previous similar message [ 2658.358797] Lustre: lustre-MDT0001: new disk, initializing [ 2658.421237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2658.427337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2660.157054] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2664.493574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2664.522716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2664.527852] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2664.615028] Lustre: lustre-OST0000: new disk, initializing [ 2664.616670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2665.845213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2665.848662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2666.040905] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2670.024088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2670.054059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.056988] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2670.114587] Lustre: lustre-OST0001: new disk, initializing [ 2670.117549] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2671.459728] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2671.933066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2671.935693] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2675.594099] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 14:36:02 (1772307362) [ 2679.375562] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 14:36:06 (1772307366) [ 2691.551812] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2691.551981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.554384] LustreError: Skipped 1 previous similar message [ 2691.558013] Lustre: Skipped 5 previous similar messages [ 2691.559879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2691.562290] Lustre: Skipped 11 previous similar messages [ 2696.749357] Lustre: server umount lustre-OST0000 complete [ 2696.751651] Lustre: Skipped 3 previous similar messages [ 2697.699226] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2697.703444] LustreError: Skipped 5 previous similar messages [ 2699.231813] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2705.857632] LustreError: 211047:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307392 with bad export cookie 2912767780256392648 [ 2705.863262] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.866258] LustreError: 211047:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2707.936285] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2707.941678] Lustre: Skipped 3 previous similar messages [ 2720.223116] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2725.075310] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2743.263105] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2745.696898] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 2751.480465] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2752.274505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2754.659466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2756.668181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2758.677140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2760.708993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2762.944098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2762.965748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.047926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2763.050431] Lustre: Skipped 1 previous similar message [ 2763.060091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2763.096686] Lustre: lustre-MDT0000: new disk, initializing [ 2763.124775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2764.488405] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2768.510687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2768.539759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.661393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2769.990183] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2773.629873] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2775.453223] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2777.570801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2777.593715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2777.596324] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2778.976518] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2779.381514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2779.386599] Lustre: Skipped 1 previous similar message [ 2779.389447] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2782.650373] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2789.857651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2789.857776] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2789.858473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2789.858478] Lustre: Skipped 10 previous similar messages [ 2789.860849] Lustre: Skipped 1 previous similar message [ 2799.496962] LustreError: 214875:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307486 with bad export cookie 2912767780256394083 [ 2799.499901] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2799.502789] LustreError: 214875:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2804.384617] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2805.326154] Key type lgssc unregistered [ 2805.440339] LNet: 218734:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2805.443062] LNet: Removed LNI 192.168.202.138@tcp [ 2805.722486] Key type .llcrypt unregistered [ 2805.723534] Key type ._llcrypt unregistered [ 2815.399351] alg: No test for adler32 (adler32-zlib) [ 2816.150351] Key type ._llcrypt registered [ 2816.151366] Key type .llcrypt registered [ 2816.184838] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2816.598988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2816.736397] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 2816.806819] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2816.808331] LNet: Accept secure, port 988 [ 2818.383092] Key type lgssc registered [ 2818.680875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2821.151452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.288735] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.301617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.622305] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2826.297510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.407585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2827.749445] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2830.190476] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2832.027721] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2834.148829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2834.247772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2835.496011] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2837.349539] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2837.919055] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2838.243855] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 2841.441589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2841.441739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.943620] Lustre: server umount lustre-OST0000 complete [ 2846.688869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.690127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2846.696972] Lustre: Skipped 1 previous similar message [ 2846.701196] Lustre: Skipped 1 previous similar message [ 2851.808355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2851.810685] Lustre: Skipped 2 previous similar messages [ 2852.303506] Lustre: server umount lustre-MDT0000 complete [ 2853.571985] LustreError: 221029:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307540 with bad export cookie 9889653487464605447 [ 2853.574949] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2853.576535] LustreError: 221029:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2853.684163] Lustre: server umount lustre-MDT0001 complete [ 2857.500260] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2858.562293] Key type lgssc unregistered [ 2858.692411] LNet: 223946:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2858.696436] LNet: Removed LNI 192.168.202.138@tcp [ 2859.006218] Key type .llcrypt unregistered [ 2859.007903] Key type ._llcrypt unregistered [ 2865.088924] alg: No test for adler32 (adler32-zlib) [ 2865.840315] Key type ._llcrypt registered [ 2865.841211] Key type .llcrypt registered [ 2865.876968] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2866.278147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2866.424301] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 2866.505079] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2866.507221] LNet: Accept secure, port 988 [ 2868.087168] Key type lgssc registered [ 2868.452895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2871.411341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2888.927378] LustreError: 225185:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 1: rc = -5 [ 2919.647191] LustreError: 225185:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 4: rc = -110 [ 2949.343227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.378813] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 2954.747764] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2955.449955] Lustre: server umount lustre-OST0000 complete [ 2961.996938] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2962.935671] Key type lgssc unregistered [ 2963.038341] LNet: 226831:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2963.041247] LNet: Removed LNI 192.168.202.138@tcp [ 2963.361490] Key type .llcrypt unregistered [ 2963.363849] Key type ._llcrypt unregistered [ 2981.467136] alg: No test for adler32 (adler32-zlib) [ 2982.219304] Key type ._llcrypt registered [ 2982.220261] Key type .llcrypt registered [ 2982.255094] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2982.581327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2982.778511] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 2982.846296] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2982.847727] LNet: Accept secure, port 988 [ 2984.423146] Key type lgssc registered [ 2985.004327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2991.178196] loop0: detected capacity change from 0 to 204800000 [ 2991.260924] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2991.268609] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2991.487908] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2991.500799] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2991.544267] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2991.579569] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2991.613410] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2991.662430] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2991.707496] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2991.747879] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2992.129181] loop0: detected capacity change from 0 to 204800000 [ 2992.186292] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2992.577918] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2992.985192] loop0: detected capacity change from 0 to 204800000 [ 2993.206733] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2993.658059] loop0: detected capacity change from 0 to 204800000 [ 2993.880782] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2994.268340] loop0: detected capacity change from 0 to 204800000 [ 2994.283451] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2995.795311] loop0: detected capacity change from 0 to 204800000 [ 2995.814457] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2997.548866] loop0: detected capacity change from 0 to 204800000 [ 2997.560104] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2999.348962] loop0: detected capacity change from 0 to 204800000 [ 2999.367031] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3001.149068] loop0: detected capacity change from 0 to 204800000 [ 3001.211193] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3001.245127] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3001.280777] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3001.288814] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3001.295781] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3001.302587] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3001.312478] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3001.322218] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3001.328257] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3001.334296] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3001.340623] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3001.349916] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3001.355867] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3001.361860] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3001.366993] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3001.371609] Lustre: 230284:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3010.015170] Lustre: 230563:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772307691/real 1772307691] req@00000000d01cf2df x1858399286329472/t0(0) o251->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772307697 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3010.032508] Lustre: server umount MGS complete [ 3010.487621] loop0: detected capacity change from 0 to 204800000 [ 3010.536556] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.701534] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3010.720793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.739731] LustreError: 230617:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 3011.092354] loop1: detected capacity change from 0 to 204800000 [ 3011.176789] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3011.195919] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.218404] Lustre: 230733:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3011.222252] Lustre: 230733:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3011.225756] Lustre: 230733:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3011.228976] Lustre: 230733:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3011.232404] Lustre: 230733:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3011.238531] Lustre: 230733:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3011.244213] Lustre: 230733:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3011.253836] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3011.322665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3011.349582] LustreError: 230733:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0001: Aborting recovery for device [ 3011.703377] loop2: detected capacity change from 0 to 204800000 [ 3011.745630] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3011.782124] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3011.784550] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 3011.801502] Lustre: 230818:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3011.804438] Lustre: 230818:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3011.807299] Lustre: 230818:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3011.810221] Lustre: 230818:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3011.813186] Lustre: 230818:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3011.816406] Lustre: 230818:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3011.819481] Lustre: 230818:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3011.872154] Lustre: Found index 0 for lustre-OST0000, updating log [ 3011.900078] LustreError: 230818:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 3012.284986] loop3: detected capacity change from 0 to 204800000 [ 3012.362620] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3012.384141] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3012.386585] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 3012.401535] Lustre: 230904:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3012.404589] Lustre: 230904:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3012.407903] Lustre: 230904:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3012.411272] Lustre: 230904:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3012.414675] Lustre: 230904:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3012.418313] Lustre: 230904:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3012.421763] Lustre: 230904:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3012.455420] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3012.457372] Lustre: Skipped 1 previous similar message [ 3012.458292] LustreError: 230676:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 3012.462599] LustreError: 230904:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 3015.459462] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 3015.467824] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 3016.481755] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 3016.482084] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 3018.032687] LustreError: 231143:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 292us: rc = -2 [ 3032.032805] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3032.033487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.044493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3032.047014] Lustre: Skipped 3 previous similar messages [ 3035.608443] Lustre: server umount lustre-MDT0000 complete [ 3035.951592] LustreError: 230624:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307723 with bad export cookie 7559687430800976993 [ 3035.953985] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3035.955983] LustreError: 230624:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3037.151732] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.152103] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3037.152466] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3037.156110] LustreError: Skipped 1 previous similar message [ 3037.160928] Lustre: Skipped 4 previous similar messages [ 3040.159900] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3040.160624] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.164271] Lustre: Skipped 2 previous similar messages [ 3040.172547] LustreError: Skipped 1 previous similar message [ 3042.222166] Lustre: server umount lustre-MDT0001 complete [ 3048.818224] Lustre: server umount lustre-OST0000 complete [ 3055.428039] Lustre: server umount lustre-OST0001 complete [ 3059.091772] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 14:42:26 (1772307746) [ 3071.720347] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 3075.025472] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3079.214883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3082.245161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3085.232446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3088.226595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3091.380504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3091.418336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3091.537643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3091.555441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3091.608793] Lustre: lustre-MDT0000: new disk, initializing [ 3091.633026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3091.639400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3093.050899] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3097.844152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3097.865173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3097.898883] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3097.921026] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3097.922505] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3097.922798] Lustre: Skipped 1 previous similar message [ 3097.961532] Lustre: lustre-MDT0001: new disk, initializing [ 3097.991071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3098.005531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3098.010535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3099.518391] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3104.364191] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3107.012426] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3109.990989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3110.036377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.042298] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3110.155944] Lustre: lustre-OST0000: new disk, initializing [ 3110.158319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3110.189295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.743271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3111.747088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3111.834881] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3115.820906] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3118.695065] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3122.647426] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3130.748974] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3173.344821] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3173.345724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.347651] LustreError: Skipped 1 previous similar message [ 3173.348453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3173.354319] Lustre: Skipped 1 previous similar message [ 3173.360025] Lustre: Skipped 1 previous similar message [ 3175.561953] Lustre: server umount lustre-OST0000 complete [ 3178.466165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3178.473958] Lustre: Skipped 1 previous similar message [ 3183.585341] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3183.591906] LustreError: Skipped 1 previous similar message [ 3185.095722] LustreError: 235075:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307872 with bad export cookie 7559687430800979674 [ 3185.099716] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3185.102131] LustreError: 235075:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3189.505499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.654700] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3200.479199] Lustre: 240566:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772307881/real 1772307881] req@00000000a1c8840d x1858399286390656/t0(0) o251->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772307887 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3200.506347] Lustre: server umount MGS complete [ 3200.508988] Lustre: Skipped 2 previous similar messages [ 3205.891434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3206.090333] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.098755] LustreError: 241200:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3206.129621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3207.867627] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3211.636108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.565943] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3216.840223] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3219.575135] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3222.891257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.870153] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3228.152933] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3261.923902] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3261.924443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3261.926755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3261.926766] Lustre: Skipped 2 previous similar messages [ 3261.929146] LustreError: Skipped 1 previous similar message [ 3261.937158] Lustre: Skipped 1 previous similar message [ 3263.791434] Lustre: server umount lustre-OST0000 complete [ 3272.160788] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.167553] LustreError: Skipped 2 previous similar messages [ 3273.495978] LustreError: 241168:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307960 with bad export cookie 7559687430800981473 [ 3273.501491] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3273.504248] LustreError: 241168:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3279.573128] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3280.950414] Key type lgssc unregistered [ 3281.101470] LNet: 245326:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3281.107112] LNet: Removed LNI 192.168.202.138@tcp [ 3281.561394] Key type .llcrypt unregistered [ 3281.563310] Key type ._llcrypt unregistered [ 3299.852667] alg: No test for adler32 (adler32-zlib) [ 3300.605343] Key type ._llcrypt registered [ 3300.606318] Key type .llcrypt registered [ 3300.644657] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 3307.407895] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3308.014455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3308.177961] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3308.258118] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3308.259704] LNet: Accept secure, port 988 [ 3309.847213] Key type lgssc registered [ 3310.423453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3314.823124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3317.681792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3320.532735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3323.425670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3326.721617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3326.757613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.861635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3327.873814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3327.926446] Lustre: lustre-MDT0000: new disk, initializing [ 3327.953517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3327.964131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3329.848839] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3335.101054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3335.126815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.160326] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3335.177683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3335.181481] Lustre: Skipped 1 previous similar message [ 3335.230957] Lustre: lustre-MDT0001: new disk, initializing [ 3335.264952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3335.279670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3335.283900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3336.887085] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3341.145503] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3342.943719] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3345.066904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3345.093908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.096646] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3345.187873] Lustre: lustre-OST0000: new disk, initializing [ 3345.189577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3345.213621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.681456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3345.684636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3346.794286] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3351.063201] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3355.111984] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3366.444942] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3372.872748] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3422.690977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.692981] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3422.693700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3422.703355] Lustre: Skipped 1 previous similar message [ 3423.729787] Lustre: server umount lustre-OST0000 complete [ 3427.296520] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3427.301690] LustreError: Skipped 1 previous similar message [ 3427.304658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.313463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.317309] Lustre: Skipped 1 previous similar message [ 3431.593279] Lustre: server umount lustre-MDT0000 complete [ 3432.928841] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.937662] LustreError: Skipped 1 previous similar message [ 3433.353901] LustreError: 250259:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308120 with bad export cookie 2905157071749850156 [ 3433.357839] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3433.361604] LustreError: 250259:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3433.552766] Lustre: server umount lustre-MDT0001 complete [ 3437.432641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.326111] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3447.775191] Lustre: 255034:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772308128/real 1772308128] req@00000000957f5924 x1858399628208512/t0(0) o251->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772308134 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3447.798162] Lustre: server umount MGS complete [ 3452.930093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.186622] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3453.198270] LustreError: 255668:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3453.235206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.261325] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3458.528491] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.294272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3459.423052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3461.239822] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3464.285747] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3466.504738] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3469.195283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3469.356674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.117610] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3474.120607] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3509.218388] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3509.219667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.231600] Lustre: Skipped 1 previous similar message [ 3509.235286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.239521] Lustre: Skipped 2 previous similar messages [ 3511.281715] Lustre: server umount lustre-OST0000 complete [ 3514.337616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3514.338675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3514.342771] Lustre: Skipped 2 previous similar messages [ 3514.345886] Lustre: Skipped 1 previous similar message [ 3516.389413] LustreError: 258472:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308203 with bad export cookie 2905157071749851955 [ 3516.393146] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3516.394077] LustreError: 258472:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3521.554335] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3522.868234] Key type lgssc unregistered [ 3523.011409] LNet: 259789:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3523.018871] LNet: Removed LNI 192.168.202.138@tcp [ 3523.472985] Key type .llcrypt unregistered [ 3523.475064] Key type ._llcrypt unregistered [ 3535.072821] alg: No test for adler32 (adler32-zlib) [ 3535.826428] Key type ._llcrypt registered [ 3535.827663] Key type .llcrypt registered [ 3535.868984] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3536.203537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3536.421429] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3536.489510] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3536.491184] LNet: Accept secure, port 988 [ 3538.071096] Key type lgssc registered [ 3538.640711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3542.369213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.543705] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3543.563097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.355806] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3548.641173] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3549.207912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.386786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3550.945877] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3553.421575] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3555.443401] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3557.871081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3558.011621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3559.364455] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3561.592475] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3563.692117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3563.715265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3563.717547] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3563.761648] Lustre: lustre-OST0001: new disk, initializing [ 3563.763385] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3563.787502] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3565.038911] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3566.982743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3566.985706] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3567.000871] LustreError: 264733:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 3567.003740] LustreError: 264733:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 3568.034057] LustreError: 264733:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 3568.040122] LustreError: 264733:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 3569.057608] LustreError: 264733:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 3569.060400] LustreError: 264733:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 3569.760336] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3569.868730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3571.105475] LustreError: 264740:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3571.107967] LustreError: 264740:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 3571.716443] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9dfc4444a800.ost_server_uuid 40 [ 3572.225493] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9dfc4444a800.ost_server_uuid in FULL state after 0 sec [ 3574.043954] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9dfc4444a800.ost_server_uuid 40 [ 3574.485314] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9dfc4444a800.ost_server_uuid in FULL state after 0 sec [ 3587.558122] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.560499] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3587.565960] Lustre: Skipped 1 previous similar message [ 3588.715202] Lustre: lustre-OST0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 3588.719962] Lustre: Skipped 1 previous similar message [ 3592.235122] Lustre: server umount lustre-OST0001 complete [ 3592.676503] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3592.680288] LustreError: Skipped 1 previous similar message [ 3595.232065] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3595.234877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.238852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3595.744860] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3597.793058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3597.793633] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.794896] Lustre: Skipped 2 previous similar messages [ 3597.801045] LustreError: Skipped 1 previous similar message [ 3599.913534] Lustre: server umount lustre-OST0000 complete [ 3602.915113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.916495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3602.924260] Lustre: Skipped 2 previous similar messages [ 3602.930602] Lustre: Skipped 1 previous similar message [ 3607.243098] Lustre: server umount lustre-MDT0000 complete [ 3608.033308] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3608.036910] LustreError: Skipped 1 previous similar message [ 3608.465589] LustreError: 261335:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308295 with bad export cookie 10227714123403261472 [ 3608.467427] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3608.471397] LustreError: 261335:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3613.308199] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3614.314194] Key type lgssc unregistered [ 3614.433449] LNet: 267006:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3614.435854] LNet: Removed LNI 192.168.202.138@tcp [ 3614.719187] Key type .llcrypt unregistered [ 3614.720207] Key type ._llcrypt unregistered [ 3627.584861] alg: No test for adler32 (adler32-zlib) [ 3628.336297] Key type ._llcrypt registered [ 3628.337180] Key type .llcrypt registered [ 3628.374791] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3628.874384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3629.071852] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3629.147701] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3629.149618] LNet: Accept secure, port 988 [ 3630.735084] Key type lgssc registered [ 3631.049425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3633.447783] loop0: detected capacity change from 0 to 1099511627776 [ 3633.593873] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 3633.597259] *** VALIDATE xfs *** [ 3633.601232] XFS (loop0): Mounting V5 Filesystem [ 3633.608668] XFS (loop0): Ending clean mount [ 3633.990206] loop1: detected capacity change from 0 to 18691697672192 [ 3649.005129] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3654.067617] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 3656.643926] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3660.063251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3662.108138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3663.978840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3665.866087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3668.453022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3668.484033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3669.579247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3669.595344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3669.650496] Lustre: lustre-MDT0000: new disk, initializing [ 3669.674182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3669.681656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3671.715073] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3677.433093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3677.477144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.528772] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3677.559975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3677.562532] Lustre: Skipped 1 previous similar message [ 3677.607560] Lustre: lustre-MDT0001: new disk, initializing [ 3677.638634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3677.649180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3677.653118] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3679.375523] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3684.238251] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3686.602369] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3689.175376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3689.215782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3689.221575] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3689.346928] Lustre: lustre-OST0000: new disk, initializing [ 3689.350563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3689.382298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.346800] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3696.137922] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3698.071127] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3699.717993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3699.724467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3700.274261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3702.104975] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3702.168518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3704.801324] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3704.801659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.804777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3704.809372] Lustre: Skipped 1 previous similar message [ 3704.813405] Lustre: Skipped 1 previous similar message [ 3708.880418] Lustre: server umount lustre-OST0000 complete [ 3713.503696] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3713.505623] LustreError: Skipped 1 previous similar message [ 3713.506713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.509876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3715.040058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3715.040307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3716.159648] Lustre: server umount lustre-MDT0000 complete [ 3717.304960] LustreError: 271937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308404 with bad export cookie 12244498447905839872 [ 3717.307601] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3717.307987] LustreError: 271937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3717.421211] Lustre: server umount lustre-MDT0001 complete [ 3719.861047] XFS (loop0): Unmounting Filesystem [ 3721.401320] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 14:53:28 (1772308408) [ 3723.462160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3723.617803] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3723.639654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3724.904586] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3727.464392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.876119] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3731.073994] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3732.858234] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3735.708373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3735.835552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.838563] Lustre: Skipped 1 previous similar message [ 3737.654167] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3740.665240] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3752.419761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.419967] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3752.428555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3752.436921] Lustre: Skipped 2 previous similar messages [ 3755.750747] Lustre: server umount lustre-OST0000 complete [ 3757.536754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.537714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3757.548257] Lustre: Skipped 2 previous similar messages [ 3763.614615] Lustre: server umount lustre-MDT0000 complete [ 3765.492742] LustreError: 276635:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308452 with bad export cookie 12244498447905840712 [ 3765.497481] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3765.502789] LustreError: 276635:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3771.629417] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3772.777255] Key type lgssc unregistered [ 3772.879307] LNet: 280405:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3772.881762] LNet: Removed LNI 192.168.202.138@tcp [ 3773.272605] Key type .llcrypt unregistered [ 3773.275043] Key type ._llcrypt unregistered [ 3784.603474] alg: No test for adler32 (adler32-zlib) [ 3785.354312] Key type ._llcrypt registered [ 3785.355360] Key type .llcrypt registered [ 3785.392511] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3785.777366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3785.932466] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3786.003637] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3786.005355] LNet: Accept secure, port 988 [ 3787.583119] Key type lgssc registered [ 3787.889491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3791.338561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3792.512563] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3792.525457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3794.262696] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3797.984672] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3798.017300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3798.155984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3799.787460] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3802.658621] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3804.999930] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3808.079810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3808.225195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.970225] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3813.111321] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3818.465462] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.466330] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3818.468577] Lustre: Skipped 1 previous similar message [ 3818.473166] Lustre: Skipped 1 previous similar message [ 3819.619280] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 3822.720769] Lustre: server umount lustre-MDT0001 complete [ 3823.582462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3823.584951] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3825.789622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3825.819656] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3825.822095] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3825.835856] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3825.926993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3827.167239] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 3831.265851] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3831.273020] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 3831.274806] Lustre: Skipped 1 previous similar message [ 3832.291812] LustreError: 285880:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 3832.294271] LustreError: 285880:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 3832.296281] LustreError: 285880:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 3832.298627] LustreError: 282010:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3832.312929] LustreError: 285885:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.138@tcp: cfg command failed: rc = -17 [ 3832.315691] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3832.315691] [ 4131.940255] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 4133.345054] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4133.345716] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.426924] Lustre: server umount lustre-MDT0001 complete [ 4136.387266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4137.052261] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.058408] LustreError: Skipped 2 previous similar messages [ 4139.021906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4139.061316] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4139.063902] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4139.079199] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4139.186664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4140.665550] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 4144.612237] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4144.620308] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 4144.622362] Lustre: Skipped 1 previous similar message [ 4146.661531] LustreError: 298144:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 4146.666062] LustreError: 298144:0:(genops.c:522:class_register_device()) Skipped 1 previous similar message [ 4146.669090] LustreError: 298144:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4146.672881] LustreError: 298144:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4146.676572] LustreError: 282010:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 4146.678574] LustreError: 282010:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 4146.697060] LustreError: 298147:0:(obd_config.c:680:class_setup()) Device 8 already setup (type osp) [ 4146.700824] LustreError: 298147:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.138@tcp: cfg command failed: rc = -17 [ 4146.705637] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.138@tcp [ 4146.705637] [ 4444.258490] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 4444.261585] Lustre: Skipped 2 previous similar messages [ 4446.688598] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4446.688747] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.697252] Lustre: Skipped 1 previous similar message [ 4449.027418] Lustre: server umount lustre-MDT0001 complete [ 4449.372369] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4449.376622] LustreError: Skipped 2 previous similar messages [ 4450.019644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4452.378782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.412424] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4452.413804] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4452.428760] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4452.534196] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4453.850684] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 4457.953978] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4457.963383] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 4457.965239] Lustre: Skipped 1 previous similar message [ 4460.004605] LustreError: 311000:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 4460.009344] LustreError: 311000:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4460.012507] LustreError: 311000:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4460.015202] LustreError: 282010:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 4460.016875] LustreError: 282010:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 4460.030732] LustreError: 311003:0:(lod_lov.c:228:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 4460.035836] LustreError: 311003:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.138@tcp: cfg command failed: rc = -114 [ 4460.038593] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 4460.038593] [ 4770.272259] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4770.273181] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4770.276301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4770.280286] Lustre: Skipped 3 previous similar messages [ 4770.283152] Lustre: Skipped 4 previous similar messages [ 4772.867765] Lustre: server umount lustre-MDT0000 complete [ 4773.969973] LustreError: 282000:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309461 with bad export cookie 8472530933706658993 [ 4773.972553] LustreError: 282000:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4773.973070] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4775.391845] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4775.391938] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4775.396313] LustreError: Skipped 2 previous similar messages [ 4778.399832] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4778.403014] Lustre: Skipped 1 previous similar message [ 4780.139581] Lustre: server umount lustre-MDT0001 complete [ 4784.181635] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 4786.807996] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4789.791667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4791.742199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4793.614051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4795.531545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4797.539234] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 15:11:24 (1772309484) [ 4799.570403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4799.590597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4799.665519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4799.675282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4799.708760] Lustre: lustre-MDT0000: new disk, initializing [ 4799.731348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4799.738114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4800.995790] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 4804.841052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4804.861733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4804.890410] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4804.905940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4804.907623] Lustre: Skipped 1 previous similar message [ 4804.939803] Lustre: lustre-MDT0001: new disk, initializing [ 4804.963821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4804.973523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4804.976502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4806.189803] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 4809.589502] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4811.231034] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4813.307593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4813.326576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4813.329401] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4813.403810] Lustre: lustre-OST0000: new disk, initializing [ 4813.405863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4813.428906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4814.649594] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all 8 [ 4818.066458] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4818.928519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4818.930951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4829.152541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4829.153532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4829.156588] Lustre: Skipped 1 previous similar message [ 4831.268734] Lustre: server umount lustre-MDT0000 complete [ 4831.270182] Lustre: Skipped 1 previous similar message [ 4832.477830] LustreError: 327881:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309519 with bad export cookie 8472530933706660890 [ 4832.479975] LustreError: 326662:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 4832.480732] LustreError: 166-1: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4832.480799] LustreError: 327881:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4832.484326] LustreError: 326662:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 4832.492934] LustreError: 326662:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 4839.903159] Lustre: 329053:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772309520/real 1772309520] req@0000000056975306 x1858400129558400/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772309526 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4848.464472] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 4851.129853] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4854.565930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4857.304823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4859.244641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4861.385825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4861.954850] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4746 sec ========== 15:12:28 (1772309548) [ 4875.868851] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 4876.877518] Key type lgssc unregistered [ 4877.005347] LNet: 334150:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4877.007872] LNet: Removed LNI 192.168.202.138@tcp [ 4877.297622] Key type .llcrypt unregistered [ 4877.299312] Key type ._llcrypt unregistered