[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 599183802 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003227] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009128] LSM: Security Framework initializing [ 0.010042] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013077] *** VALIDATE selinux *** [ 0.019627] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024211] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] *** VALIDATE tmpfs *** [ 0.029423] *** VALIDATE proc *** [ 0.031223] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.034233] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037022] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.040762] debug: unmapping init [mem 0xffffffff98e59000-0xffffffff98e60fff] [ 0.042128] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043647] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044019] ... version: 2 [ 0.045011] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047009] ... value mask: 0000ffffffffffff [ 0.048010] ... max period: 00007fffffffffff [ 0.049013] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051270] rcu: Hierarchical SRCU implementation. [ 0.053362] smp: Bringing up secondary CPUs ... [ 0.054541] x86: Booting SMP configuration: [ 0.055019] .... node #0, CPUs: #1 #2 #3 [ 0.070085] smp: Brought up 1 node, 4 CPUs [ 0.072018] smpboot: Max logical packages: 1 [ 0.073019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.197565] node 0 deferred pages initialised in 122ms [ 0.202111] devtmpfs: initialized [ 0.208331] x86/mm: Memory block size: 128MB [ 0.212000] gcov: version magic: 0x41383552 [ 0.215416] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.218093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.220314] pinctrl core: initialized pinctrl subsystem [ 0.221114] [ 0.221470] ************************************************************* [ 0.223010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.225007] ** ** [ 0.226008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.228009] ** ** [ 0.230012] ** This means that this kernel is built to expose internal ** [ 0.232008] ** IOMMU data structures, which may compromise security on ** [ 0.233011] ** your system. ** [ 0.235007] ** ** [ 0.236006] ** If you see this message and you are not debugging the ** [ 0.238013] ** kernel, report this immediately to your vendor! ** [ 0.239056] ** ** [ 0.241007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.242000] ************************************************************* [ 0.244051] NET: Registered protocol family 16 [ 0.245500] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.248055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.250091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.253544] cpuidle: using governor menu [ 0.254724] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.256432] PCI: Using configuration type 1 for base access [ 0.257137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.265391] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.266020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.268694] cryptd: max_cpu_qlen set to 1000 [ 0.274169] ACPI: Added _OSI(Module Device) [ 0.276012] ACPI: Added _OSI(Processor Device) [ 0.277011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.279014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.282923] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.290589] ACPI: Interpreter enabled [ 0.291000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.291000] ACPI: Using IOAPIC for interrupt routing [ 0.291119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.294502] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.302483] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.304028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.306013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.308080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.312335] acpiphp: Slot [2] registered [ 0.314124] acpiphp: Slot [3] registered [ 0.315072] acpiphp: Slot [4] registered [ 0.316084] acpiphp: Slot [5] registered [ 0.317106] acpiphp: Slot [6] registered [ 0.318086] acpiphp: Slot [7] registered [ 0.319091] acpiphp: Slot [8] registered [ 0.320078] acpiphp: Slot [9] registered [ 0.321058] acpiphp: Slot [10] registered [ 0.322085] acpiphp: Slot [11] registered [ 0.323053] acpiphp: Slot [12] registered [ 0.324046] acpiphp: Slot [13] registered [ 0.326074] acpiphp: Slot [14] registered [ 0.327059] acpiphp: Slot [15] registered [ 0.328066] acpiphp: Slot [16] registered [ 0.329064] acpiphp: Slot [17] registered [ 0.330094] acpiphp: Slot [18] registered [ 0.331067] acpiphp: Slot [19] registered [ 0.332059] acpiphp: Slot [20] registered [ 0.333069] acpiphp: Slot [21] registered [ 0.335091] acpiphp: Slot [22] registered [ 0.336063] acpiphp: Slot [23] registered [ 0.337104] acpiphp: Slot [24] registered [ 0.338093] acpiphp: Slot [25] registered [ 0.339059] acpiphp: Slot [26] registered [ 0.340062] acpiphp: Slot [27] registered [ 0.341061] acpiphp: Slot [28] registered [ 0.342313] acpiphp: Slot [29] registered [ 0.343094] acpiphp: Slot [30] registered [ 0.344084] acpiphp: Slot [31] registered [ 0.345045] PCI host bridge to bus 0000:00 [ 0.346015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.348014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.350016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.352017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.353016] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.355021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.357168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.360278] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.364325] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.373017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.377049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.380015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.381020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.383010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.385404] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.387790] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.390027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.391714] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395896] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.402830] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.406788] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.412700] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.420015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.426017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.439015] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.441000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.451017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.465022] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.477695] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.491027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.493000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.502019] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.516992] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.522000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.532034] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.551032] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.570000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.582020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.590023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.610028] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.624000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.629020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.640024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.657024] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.674000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.676367] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.678361] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.679465] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.682374] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.686318] iommu: Default domain type: Passthrough [ 0.687512] SCSI subsystem initialized [ 0.689196] ACPI: bus type USB registered [ 0.690105] usbcore: registered new interface driver usbfs [ 0.692120] usbcore: registered new interface driver hub [ 0.694132] usbcore: registered new device driver usb [ 0.695189] pps_core: LinuxPPS API ver. 1 registered [ 0.697015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.700082] PTP clock support registered [ 0.702064] EDAC MC: Ver: 3.0.0 [ 0.704442] PCI: Using ACPI for IRQ routing [ 0.707622] NetLabel: Initializing [ 0.709021] NetLabel: domain hash size = 128 [ 0.710009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.712104] NetLabel: unlabeled traffic allowed by default [ 0.714138] vgaarb: loaded [ 0.716692] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.718013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.726000] clocksource: Switched to clocksource kvm-clock [ 0.866284] VFS: Disk quotas dquot_6.6.0 [ 0.867657] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.870113] *** VALIDATE ramfs *** [ 0.871236] *** VALIDATE hugetlbfs *** [ 0.872788] pnp: PnP ACPI init [ 0.875132] pnp: PnP ACPI: found 6 devices [ 0.890582] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.893360] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.895270] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.897158] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.899236] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.901419] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.904385] NET: Registered protocol family 2 [ 0.906407] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.911425] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.914178] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.919527] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.922447] TCP: Hash tables configured (established 65536 bind 65536) [ 0.924921] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.927559] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.929853] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.932427] NET: Registered protocol family 1 [ 0.936741] RPC: Registered named UNIX socket transport module. [ 0.939379] RPC: Registered udp transport module. [ 0.940668] RPC: Registered tcp transport module. [ 0.941971] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.943902] NET: Registered protocol family 44 [ 0.945152] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.946890] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.948607] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.950443] PCI: CLS 0 bytes, default 64 [ 0.951854] Unpacking initramfs... [ 2.639823] debug: unmapping init [mem 0xffff91a0bcc54000-0xffff91a0bffbffff] [ 2.647176] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.649290] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.653406] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.202468] Initialise system trusted keyrings [ 3.204203] Key type blacklist registered [ 3.205963] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.215197] zbud: loaded [ 3.221650] *** VALIDATE nfs *** [ 3.223047] *** VALIDATE nfs4 *** [ 3.224404] pstore: using deflate compression [ 3.227971] Platform Keyring initialized [ 3.470190] NET: Registered protocol family 38 [ 3.473790] Key type asymmetric registered [ 3.475348] Asymmetric key parser 'x509' registered [ 3.477215] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.481350] io scheduler mq-deadline registered [ 3.483124] io scheduler kyber registered [ 3.484917] io scheduler bfq registered [ 3.488800] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.496464] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.499509] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.505990] ACPI: Power Button [PWRF] [ 3.622196] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.748712] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.987582] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.087032] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.358490] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.404699] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.440049] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.444652] Non-volatile memory driver v1.3 [ 4.446054] Linux agpgart interface v0.103 [ 4.479770] virtio_blk virtio1: [vda] 67976 512-byte logical blocks (34.8 MB/33.2 MiB) [ 4.483434] vda: detected capacity change from 0 to 34803712 [ 4.513246] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.516241] vdb: detected capacity change from 0 to 1073741824 [ 4.535500] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.538296] vdc: detected capacity change from 0 to 2621440000 [ 4.562247] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.564401] vdd: detected capacity change from 0 to 2621440000 [ 4.581318] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.583862] vde: detected capacity change from 0 to 4294967296 [ 4.601240] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.603908] vdf: detected capacity change from 0 to 4294967296 [ 4.613805] libphy: Fixed MDIO Bus: probed [ 4.634655] usbcore: registered new interface driver usbserial_generic [ 4.636561] usbserial: USB Serial support registered for generic [ 4.638864] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.645913] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.648227] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.653354] mousedev: PS/2 mouse device common for all mice [ 4.656528] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.662096] rtc_cmos 00:05: RTC can wake from S4 [ 4.665638] rtc_cmos 00:05: registered as rtc0 [ 4.668402] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.668535] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.674256] intel_pstate: CPU model not supported [ 4.681771] hid: raw HID events driver (C) Jiri Kosina [ 4.684087] usbcore: registered new interface driver usbhid [ 4.694103] usbhid: USB HID core driver [ 4.695802] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.695846] drop_monitor: Initializing network drop monitor service [ 4.706461] Initializing XFRM netlink socket [ 4.710942] NET: Registered protocol family 10 [ 4.716911] Segment Routing with IPv6 [ 4.718193] NET: Registered protocol family 17 [ 4.720661] mpls_gso: MPLS GSO support [ 4.731385] RAS: Correctable Errors collector initialized. [ 4.733050] AVX version of gcm_enc/dec engaged. [ 4.734992] AES CTR mode by8 optimization enabled [ 4.867398] sched_clock: Marking stable (4867376679, 0)->(5905950857, -1038574178) [ 4.870849] registered taskstats version 1 [ 4.875660] Loading compiled-in X.509 certificates [ 4.885575] zswap: loaded using pool lzo/zbud [ 4.936275] Key type big_key registered [ 4.959102] Key type encrypted registered [ 4.962304] ima: No TPM chip found, activating TPM-bypass! [ 4.967776] ima: Allocated hash algorithm: sha1 [ 4.969277] ima: No architecture policies found [ 4.970661] evm: Initialising EVM extended attributes: [ 4.972329] evm: security.selinux [ 4.973712] evm: security.ima [ 4.975151] evm: security.capability [ 4.976664] evm: HMAC attrs: 0x1 [ 4.980894] rtc_cmos 00:05: setting system clock to 2025-10-22 08:20:21 UTC (1761121221) [ 4.990618] debug: unmapping init [mem 0xffffffff99e03000-0xffffffff99ffffff] [ 4.993936] debug: unmapping init [mem 0xffffffff98b82000-0xffffffff98e58fff] [ 5.002184] Write protecting the kernel read-only data: 28672k [ 5.007645] debug: unmapping init [mem 0xffffffff97203000-0xffffffff973fffff] [ 5.011151] debug: unmapping init [mem 0xffffffff97b14000-0xffffffff97bfffff] [ 5.066425] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.076701] systemd[1]: Detected virtualization kvm. [ 5.078780] systemd[1]: Detected architecture x86-64. [ 5.081323] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.108573] systemd[1]: No hostname configured. [ 5.111162] systemd[1]: Set hostname to . [ 5.114427] random: systemd: uninitialized urandom read (16 bytes read) [ 5.116311] systemd[1]: Initializing machine ID from random generator. [ 5.171504] random: ln: uninitialized urandom read (6 bytes read) [ 5.328436] random: systemd: uninitialized urandom read (16 bytes read) [ 5.330771] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.337908] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.342869] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.597887] device-mapper: uevent: version 1.0.3 [ 6.599965] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.785437] random: fast init done Starting dracut initqueue hook... [ 7.851908] virtio_net virtio0 ens2: renamed from eth0 [ 8.125388] scsi host0: ata_piix [ 8.232820] scsi host1: ata_piix [ 8.234124] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.248698] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.490291] random: crng init done [ 13.494812] random: 7 urandom warning(s) missed due to ratelimiting [ 14.372334] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.521332] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.414830] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.930969] SELinux: Disabled at runtime. [ 18.033818] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.053430] systemd[1]: Detected virtualization kvm. [ 18.055357] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.241349] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.248727] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.258979] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.267458] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.273957] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.284968] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.292362] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Reached target Paths. [ OK ] Started Jour[ 19.580765] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS nal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.051755] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.579143] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.713342] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.999530] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.052483] EDAC sbridge: Ver: 1.1.2 [ 24.589340] Key type dns_resolver registered [ 25.194158] NFS: Registering the id_resolver key type [ 25.201989] Key type id_resolver registered [ 25.203618] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg433-server login: [ 49.158459] libcfs: loading out-of-tree module taints kernel. [ 49.175412] alg: No test for adler32 (adler32-zlib) [ 49.927402] Key type ._llcrypt registered [ 49.928732] Key type .llcrypt registered [ 49.968284] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 56.909703] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 57.451152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.714845] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 57.994493] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 57.996539] LNet: Accept secure, port 988 [ 59.607131] Key type lgssc registered [ 60.129498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 66.511215] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 67.486806] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.922107] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 73.480846] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 76.131291] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.363189] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 86.043971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.078480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 87.169891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 87.183455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 87.228320] Lustre: lustre-MDT0000: new disk, initializing [ 87.256105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 87.264446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 88.824979] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 94.174269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.208715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 94.240437] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 94.253646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 94.256460] Lustre: Skipped 1 previous similar message [ 94.300619] Lustre: lustre-MDT0001: new disk, initializing [ 94.325751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 94.338599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 94.343349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 95.849923] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 101.129139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.154931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 101.158651] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 101.244200] Lustre: lustre-OST0000: new disk, initializing [ 101.246303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 101.267960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 102.710260] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 103.340738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 103.344083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 107.749905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.775393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 107.778648] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 107.811493] Lustre: lustre-OST0001: new disk, initializing [ 107.813414] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 107.829748] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 109.175129] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 109.995926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 109.998995] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 114.723128] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 120.484129] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 126.832650] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing check_logdir /tmp/testlogs/ [ 128.441992] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing yml_node [ 129.891280] Lustre: DEBUG MARKER: Client: 2.15.7.9 [ 130.719197] Lustre: DEBUG MARKER: MDS: 2.15.7.9 [ 131.543285] Lustre: DEBUG MARKER: OSS: 2.15.7.9 [ 132.099925] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Oct 22 04:22:28 EDT 2025 [ 135.103082] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 135.667845] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 151.007693] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 151.008981] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 151.011467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 151.015582] Lustre: Skipped 1 previous similar message [ 153.616870] Lustre: server umount lustre-MDT0000 complete [ 155.153047] LustreError: 6221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121371 with bad export cookie 17143478184431889331 [ 155.154716] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 155.158107] LustreError: 6221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 156.127778] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 156.127832] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 156.128130] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 156.128137] Lustre: Skipped 3 previous similar messages [ 156.133824] LustreError: Skipped 1 previous similar message [ 156.139212] Lustre: Skipped 3 previous similar messages [ 160.159707] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 160.160032] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 160.162115] Lustre: Skipped 2 previous similar messages [ 160.168341] LustreError: Skipped 1 previous similar message [ 161.184139] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 161.189517] LustreError: Skipped 1 previous similar message [ 161.344544] Lustre: server umount lustre-MDT0001 complete [ 177.119140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 177.136206] Lustre: server umount lustre-OST0000 complete [ 184.797395] Lustre: server umount lustre-OST0001 complete [ 187.479423] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 190.783259] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 194.789624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 197.378742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 199.828491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.299967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.161709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.202274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.312314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 205.331073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 205.375966] Lustre: lustre-MDT0000: new disk, initializing [ 205.429662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.438969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.324455] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 212.313252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 212.351348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.394245] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 212.507242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 214.282924] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 218.442388] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 220.615442] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 222.975198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 223.005248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.008416] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 223.094825] Lustre: lustre-OST0000: new disk, initializing [ 223.096159] Lustre: Skipped 1 previous similar message [ 223.097963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 223.100762] Lustre: Skipped 2 previous similar messages [ 223.125279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.127662] Lustre: Skipped 1 previous similar message [ 224.660410] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 225.085120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 225.088930] Lustre: Skipped 1 previous similar message [ 225.090859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 228.775706] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 230.414484] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 230.498727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 232.164777] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 232.251545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 235.488432] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 235.489369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 235.491967] LustreError: Skipped 1 previous similar message [ 235.492731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 235.496914] Lustre: Skipped 1 previous similar message [ 235.498789] Lustre: Skipped 2 previous similar messages [ 240.610404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 240.612856] Lustre: Skipped 2 previous similar messages [ 247.263134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 247.326205] Lustre: server umount lustre-OST0000 complete [ 250.848397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.849258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 250.853855] Lustre: Skipped 1 previous similar message [ 250.856215] Lustre: Skipped 2 previous similar messages [ 255.967956] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.973787] LustreError: Skipped 1 previous similar message [ 256.095222] LustreError: 15162:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121472 with bad export cookie 17143478184431890773 [ 256.098707] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 256.099798] LustreError: 15162:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 256.215299] Lustre: server umount lustre-MDT0001 complete [ 256.216787] Lustre: Skipped 1 previous similar message [ 257.601781] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:24:33 (1761121473) [ 258.454107] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 259.060473] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:24:35 (1761121475) [ 261.358237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 261.509593] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 261.532551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 263.074765] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 266.198094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 267.896243] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 270.609680] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 272.693435] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 275.133076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 276.849937] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 279.489082] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 286.688559] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 286.689917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.698507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 286.700751] Lustre: Skipped 1 previous similar message [ 292.312691] Lustre: server umount lustre-OST0000 complete [ 296.928659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.928724] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 296.932519] Lustre: Skipped 2 previous similar messages [ 300.916137] LustreError: 21706:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121517 with bad export cookie 17143478184431891627 [ 300.918765] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 300.921420] LustreError: 21706:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 305.510996] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 306.447922] Key type lgssc unregistered [ 306.554353] LNet: 22507:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.557240] LNet: Removed LNI 192.168.204.133@tcp [ 306.837744] Key type .llcrypt unregistered [ 306.838841] Key type ._llcrypt unregistered [ 315.620069] alg: No test for adler32 (adler32-zlib) [ 316.371549] Key type ._llcrypt registered [ 316.373376] Key type .llcrypt registered [ 316.424843] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 316.791161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 317.034130] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 317.117826] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 317.119365] LNet: Accept secure, port 988 [ 318.703116] Key type lgssc registered [ 319.094726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 321.870056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 323.013496] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.031338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 324.523992] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 327.495124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 327.613190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 329.049079] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 331.683380] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 333.741174] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 336.126370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 336.246090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 337.784950] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 339.300466] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 340.472627] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 349.664137] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 349.668085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.675176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 353.760726] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 353.761824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 353.763465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.351128] Lustre: server umount lustre-OST0000 complete [ 358.368159] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 358.371244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.376267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.377591] Lustre: Skipped 1 previous similar message [ 363.999837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.001603] Lustre: Skipped 2 previous similar messages [ 369.119916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.121583] Lustre: Skipped 1 previous similar message [ 371.167122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 371.212071] Lustre: server umount lustre-MDT0000 complete [ 372.613182] LustreError: 23962:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121589 with bad export cookie 2839100997494794194 [ 372.615695] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 372.619082] LustreError: 23962:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 372.770636] Lustre: server umount lustre-MDT0001 complete [ 377.945045] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 379.040700] Key type lgssc unregistered [ 379.164410] LNet: 27638:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.167464] LNet: Removed LNI 192.168.204.133@tcp [ 379.473899] Key type .llcrypt unregistered [ 379.475126] Key type ._llcrypt unregistered [ 389.023453] alg: No test for adler32 (adler32-zlib) [ 389.775453] Key type ._llcrypt registered [ 389.777337] Key type .llcrypt registered [ 389.821758] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 390.306738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 390.452207] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 390.534958] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 390.536930] LNet: Accept secure, port 988 [ 392.119141] Key type lgssc registered [ 392.499492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 395.415807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.572948] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.589239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.189085] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 401.273295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.400074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 402.908948] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 405.502848] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 407.637910] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 410.082724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 410.199995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.718900] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 412.264323] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 414.405729] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 421.857078] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 421.857781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 421.859784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 421.865321] Lustre: Skipped 2 previous similar messages [ 425.816226] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 425.818840] Lustre: Skipped 2 previous similar messages [ 426.978450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 426.980707] Lustre: Skipped 2 previous similar messages [ 427.585953] Lustre: server umount lustre-MDT0000 complete [ 428.753646] LustreError: 29078:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121645 with bad export cookie 13056312428630447507 [ 428.756960] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 428.758029] LustreError: 29078:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 430.927721] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.935298] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 432.096234] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 432.096400] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.168251] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.168272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 435.172450] LustreError: Skipped 1 previous similar message [ 435.176535] Lustre: Skipped 2 previous similar messages [ 440.287988] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.292868] LustreError: Skipped 2 previous similar messages [ 443.215673] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 443.218124] Lustre: Skipped 3 previous similar messages [ 443.359143] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 443.447854] Lustre: server umount lustre-MDT0001 complete [ 455.322114] Lustre: server umount lustre-OST0000 complete [ 462.041569] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 463.035371] Key type lgssc unregistered [ 463.153332] LNet: 33076:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.155698] LNet: Removed LNI 192.168.204.133@tcp [ 463.441573] Key type .llcrypt unregistered [ 463.442685] Key type ._llcrypt unregistered [ 472.526877] alg: No test for adler32 (adler32-zlib) [ 473.278373] Key type ._llcrypt registered [ 473.279751] Key type .llcrypt registered [ 473.316495] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 473.708992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 473.919961] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 474.001327] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 474.003266] LNet: Accept secure, port 988 [ 475.591124] Key type lgssc registered [ 475.930517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 478.464125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.604964] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.621902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 480.983600] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 483.773053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.888549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 485.290631] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 487.763792] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 489.628746] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 491.856995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.966560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 493.342618] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 495.078970] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 495.817590] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 504.290339] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 504.290442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.298310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 504.300700] Lustre: Skipped 1 previous similar message [ 507.225331] Lustre: lustre-OST0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 508.901458] Lustre: server umount lustre-OST0000 complete [ 524.768588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.768964] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 524.769226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 524.773427] Lustre: Skipped 2 previous similar messages [ 527.815163] Lustre: server umount lustre-MDT0000 complete [ 529.084975] LustreError: 34517:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121745 with bad export cookie 18167329789448511213 [ 529.087625] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.088037] LustreError: 34517:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 535.288939] Lustre: server umount lustre-MDT0001 complete [ 539.972498] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 541.015058] Key type lgssc unregistered [ 541.141361] LNet: 38367:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.144429] LNet: Removed LNI 192.168.204.133@tcp [ 541.432648] Key type .llcrypt unregistered [ 541.434315] Key type ._llcrypt unregistered [ 550.260407] alg: No test for adler32 (adler32-zlib) [ 551.011530] Key type ._llcrypt registered [ 551.013205] Key type .llcrypt registered [ 551.051552] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 551.391414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 551.607481] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 551.677330] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 551.678835] LNet: Accept secure, port 988 [ 553.255124] Key type lgssc registered [ 553.606656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 556.231275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.371961] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.384875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 558.752427] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 561.598443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.727525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 563.147942] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 565.583928] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 567.463248] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 569.676795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 569.780112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.128344] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 572.836747] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 573.638560] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 580.455899] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 581.600542] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 588.260114] LNetError: 120-3: Refusing connection from 192.168.204.133 for 127.0.0.2@tcp: No matching NI [ 588.262050] LNetError: 39129:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 588.264100] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 592.290841] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-22 4h30m08s [ 592.766320] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 595.285343] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 597.475525] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.475653] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 597.476317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 597.477421] LNetError: 120-3: Refusing connection from 192.168.204.133 for 127.0.0.2@tcp: No matching NI [ 597.477491] LNetError: 39126:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 597.477516] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 597.478695] Lustre: Skipped 1 previous similar message [ 599.621464] Lustre: server umount lustre-MDT0000 complete [ 600.399243] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.754447] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 602.592689] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.596512] LNetError: 120-3: Refusing connection from 192.168.204.133 for 127.0.0.2@tcp: No matching NI [ 602.598885] LustreError: Skipped 2 previous similar messages [ 602.603698] LNetError: 39127:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 602.605752] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 602.998938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.038997] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 603.039490] LustreError: 11-0: MGC192.168.204.133@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 603.043708] Lustre: Evicted from MGS (at 192.168.204.133@tcp) after server handle changed from 0x739a8b56835545a9 to 0x739a8b56835549ed [ 603.049499] Lustre: MGC192.168.204.133@tcp: Connection restored to 192.168.204.133@tcp (at 0@lo) [ 603.123320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 603.152976] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 604.511953] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 605.644730] Lustre: DEBUG MARKER: Wait for df (28818) ... [ 606.183399] Lustre: DEBUG MARKER: done [ 607.711759] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 607.714310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.717824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 607.719461] Lustre: Skipped 2 previous similar messages [ 608.226091] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 608.226279] LNetError: 120-3: Refusing connection from 192.168.204.133 for 127.0.0.2@tcp: No matching NI [ 608.231720] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 608.232294] LNetError: 39128:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 608.239044] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 613.280919] Lustre: server umount lustre-OST0000 complete [ 618.464829] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 618.464887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 618.466100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 618.466111] Lustre: Skipped 2 previous similar messages [ 618.466950] LustreError: Skipped 1 previous similar message [ 618.469673] Lustre: Skipped 2 previous similar messages [ 618.471083] LNetError: 120-3: Refusing connection from 192.168.204.133 for 127.0.0.2@tcp: No matching NI [ 618.476233] LNetError: 39129:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 618.478617] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 620.679459] Lustre: server umount lustre-MDT0000 complete [ 621.900806] LustreError: 42522:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121838 with bad export cookie 8330123664462006765 [ 621.903561] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 621.904558] LustreError: 42522:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 626.585221] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 627.690718] Key type lgssc unregistered [ 627.806333] LNet: 44915:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.809061] LNet: Removed LNI 192.168.204.133@tcp [ 628.096840] Key type .llcrypt unregistered [ 628.097852] Key type ._llcrypt unregistered [ 639.185202] alg: No test for adler32 (adler32-zlib) [ 639.936407] Key type ._llcrypt registered [ 639.937534] Key type .llcrypt registered [ 639.973035] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 646.115456] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 646.536556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 646.665629] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 646.738885] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 646.740653] LNet: Accept secure, port 988 [ 648.319192] Key type lgssc registered [ 648.644380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 651.713480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 654.257885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 656.369669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 658.753453] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 661.225056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 661.249366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.333683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 662.344605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 662.380490] Lustre: lustre-MDT0000: new disk, initializing [ 662.403253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.408837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 663.807427] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 667.952564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 667.977113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.007423] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 668.022720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 668.024358] Lustre: Skipped 1 previous similar message [ 668.057629] Lustre: lustre-MDT0001: new disk, initializing [ 668.089739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 668.100545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 668.103311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 669.474479] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 673.284387] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 675.166344] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 677.426284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 677.453190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 677.455596] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 677.541376] Lustre: lustre-OST0000: new disk, initializing [ 677.543467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 677.569073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 678.949863] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 679.924751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 679.927870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 682.733749] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 684.272643] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 684.352629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 685.818580] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 685.902304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 690.144470] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 690.145018] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.146960] LustreError: Skipped 1 previous similar message [ 690.150851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 690.153780] Lustre: Skipped 1 previous similar message [ 692.629605] Lustre: server umount lustre-OST0000 complete [ 695.264436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.265122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 695.269965] Lustre: Skipped 2 previous similar messages [ 700.039860] Lustre: server umount lustre-MDT0000 complete [ 700.384265] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.388086] LustreError: Skipped 1 previous similar message [ 701.358086] LustreError: 49003:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121917 with bad export cookie 2401888857137323270 [ 701.361529] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.361752] LustreError: 49003:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 701.483450] Lustre: server umount lustre-MDT0001 complete [ 704.290627] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 04:32:00 (1761121920) [ 706.495559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 706.638032] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.660190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 708.063402] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 710.893848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.000949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 712.355793] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 714.879993] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 716.736675] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 718.923951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 719.026375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.355042] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 722.810379] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 730.264839] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-22 4h32m26s [ 730.831357] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 733.108346] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 733.896597] Lustre: DEBUG MARKER: Creating a test file and stat it [ 733.911429] LustreError: 53560:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 739.301660] LNetError: 120-3: Refusing connection from 192.168.204.133 for 127.0.0.2@tcp: No matching NI [ 739.306843] LNetError: 47226:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 739.309819] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 754.020833] Lustre: lustre-MDT0000: Client b006ecf3-a9db-4ed8-aa8c-9de43f78ff2e (at 192.168.204.33@tcp) reconnecting [ 773.951147] LustreError: 53560:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 774.683790] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 775.435717] Lustre: DEBUG MARKER: done [ 777.697650] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 777.698955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 777.704695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 777.707566] Lustre: Skipped 2 previous similar messages [ 782.816424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 782.818205] Lustre: Skipped 1 previous similar message [ 783.144685] Lustre: server umount lustre-OST0000 complete [ 787.936655] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 787.937049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 787.937484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 787.943974] Lustre: Skipped 2 previous similar messages [ 787.946386] LNetError: 120-3: Refusing connection from 192.168.204.133 for 127.0.0.2@tcp: No matching NI [ 787.948457] LNetError: 47223:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 787.950532] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 790.535165] Lustre: server umount lustre-MDT0000 complete [ 791.772346] LustreError: 53546:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761122008 with bad export cookie 2401888857137324110 [ 791.774322] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 791.778326] LustreError: 53546:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 796.371120] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 797.289746] Key type lgssc unregistered [ 797.396456] LNet: 57973:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.399644] LNet: Removed LNI 192.168.204.133@tcp [ 797.670526] Key type .llcrypt unregistered [ 797.671553] Key type ._llcrypt unregistered [ 808.679331] alg: No test for adler32 (adler32-zlib) [ 809.431299] Key type ._llcrypt registered [ 809.432404] Key type .llcrypt registered [ 809.471085] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 815.486420] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 815.863786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 816.061358] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 816.146744] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 816.149624] LNet: Accept secure, port 988 [ 817.737169] Key type lgssc registered [ 818.143802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 821.290383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 823.645220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 826.046549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 828.362422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 830.682387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 830.705894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 831.785370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 831.797763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 831.839214] Lustre: lustre-MDT0000: new disk, initializing [ 831.865245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.872486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 833.253649] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 837.204843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 837.223900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.249994] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 837.264130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 837.265739] Lustre: Skipped 1 previous similar message [ 837.294951] Lustre: lustre-MDT0001: new disk, initializing [ 837.319670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 837.330043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 837.332874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 838.641709] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 842.250383] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 844.024656] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 846.087803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 846.114131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 846.116976] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 846.211214] Lustre: lustre-OST0000: new disk, initializing [ 846.213933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 846.241885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 847.700653] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 848.118294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 848.121963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 851.616804] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 853.128160] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 853.205287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 854.613025] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 854.685149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 858.593201] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 858.593511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.594432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 858.596783] LustreError: Skipped 1 previous similar message [ 858.602023] Lustre: Skipped 1 previous similar message [ 861.397540] Lustre: server umount lustre-OST0000 complete [ 863.199808] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 863.203094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.207406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 863.209260] Lustre: Skipped 1 previous similar message [ 868.801843] Lustre: server umount lustre-MDT0000 complete [ 868.831944] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.836075] LustreError: Skipped 1 previous similar message [ 870.108988] LustreError: 64144:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761122086 with bad export cookie 13924822948941043410 [ 870.111741] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 870.112963] LustreError: 64144:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 870.236596] Lustre: server umount lustre-MDT0001 complete [ 873.157283] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 04:34:49 (1761122089) [ 873.706403] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 874.341220] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 04:34:50 (1761122090) [ 878.378347] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 879.931370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.082369] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 880.097396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 880.455084] Lustre: Failing over lustre-MDT0000 [ 880.457029] LustreError: 67072:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 880.460152] LustreError: 67072:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 880.463207] LustreError: 67072:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 886.751151] Lustre: 67123:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761122097/real 1761122097] req@00000000017a59ff x1846670295843456/t0(0) o251->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761122103 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 886.805703] Lustre: server umount lustre-MDT0000 complete [ 888.967951] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 04:35:05 (1761122105) [ 891.167594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 891.311077] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 891.326875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.801800] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 895.707624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 897.291522] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 899.898992] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 901.903268] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 904.260051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 904.377074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 904.381118] Lustre: Skipped 1 previous similar message [ 905.865983] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 908.437799] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 915.313196] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 921.058990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.061988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 921.064410] Lustre: Skipped 3 previous similar messages [ 921.069404] Lustre: Skipped 3 previous similar messages [ 923.280516] Lustre: server umount lustre-MDT0000 complete [ 924.564463] LustreError: 69641:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761122141 with bad export cookie 13924822948941044523 [ 924.568382] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 924.568859] LustreError: 69641:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 925.839841] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 927.361297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 930.265546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.232366] Lustre: Evicted from MGS (at 192.168.204.133@tcp) after server handle changed from 0xc13ee8b1f7d69b2b to 0xc13ee8b1f7d6a2b0 [ 931.237127] Lustre: MGC192.168.204.133@tcp: Connection restored to 192.168.204.133@tcp (at 0@lo) [ 931.340503] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.362642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.906486] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 933.407119] Lustre: 60287:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761122142/real 1761122142] req@000000000103c9a3 x1846670295857344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761122149 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 933.415216] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.903459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 936.047620] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 937.055162] Lustre: 60288:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761122145/real 1761122145] req@0000000097ece13d x1846670295857408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761122152 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 937.060084] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 937.518502] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 940.070466] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 942.053435] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 946.144239] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 946.146724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.151966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 950.019449] Lustre: server umount lustre-MDT0000 complete [ 950.020854] Lustre: Skipped 1 previous similar message [ 951.261739] LustreError: 67671:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761122167 with bad export cookie 13924822948941046448 [ 951.265978] LustreError: 67671:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 951.266300] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 953.632982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 956.837524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.839136] Lustre: 60287:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761122169/real 1761122169] req@00000000f8281a4d x1846670295868800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761122176 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 959.846890] Lustre: 60287:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 959.849808] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.854142] Lustre: Skipped 2 previous similar messages [ 964.063128] Lustre: 60288:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761122173/real 1761122173] req@000000007a0d0cf0 x1846670295868864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761122180 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 968.032442] Lustre: Evicted from MGS (at 192.168.204.133@tcp) after server handle changed from 0xc13ee8b1f7d6a2b0 to 0xc13ee8b1f7d6a8bb [ 968.036932] Lustre: MGC192.168.204.133@tcp: Connection restored to (at 0@lo) [ 968.039180] Lustre: Skipped 1 previous similar message [ 968.134787] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.139600] LustreError: Skipped 2 previous similar messages [ 968.159413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.161220] Lustre: Skipped 1 previous similar message [ 968.177875] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 969.611927] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 972.437142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 972.588660] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 973.959588] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 976.405530] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 976.676242] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 978.324225] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 983.007945] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 983.010060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.014865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.016617] Lustre: Skipped 3 previous similar messages [ 986.181055] Lustre: server umount lustre-MDT0000 complete [ 986.182162] Lustre: Skipped 1 previous similar message [ 987.382255] LustreError: 69641:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761122203 with bad export cookie 13924822948941047995 [ 987.386115] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 987.387526] LustreError: 69641:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 989.662270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 990.729072] Lustre: DEBUG MARKER: files compared the same [ 996.831136] Lustre: 76818:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761122207/real 1761122207] req@00000000dfe762ad x1846670295878592/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761122213 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 996.839302] Lustre: 76818:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1003.074944] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1004.129498] Key type lgssc unregistered [ 1004.242402] LNet: 77724:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.245583] LNet: Removed LNI 192.168.204.133@tcp [ 1004.516323] Key type .llcrypt unregistered [ 1004.517603] Key type ._llcrypt unregistered [ 1012.635172] alg: No test for adler32 (adler32-zlib) [ 1013.386283] Key type ._llcrypt registered [ 1013.387158] Key type .llcrypt registered [ 1013.418644] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1013.912117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1014.050998] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 1014.145492] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1014.147598] LNet: Accept secure, port 988 [ 1015.727110] Key type lgssc registered [ 1016.084608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1018.479765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1019.631455] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1019.649217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.977574] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug +malloc all 8 [ 1023.694225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.791309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1025.122510] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug +malloc all 8 [ 1027.547358] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1029.297918] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1031.414334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1031.500855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1032.803207] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug +malloc all 8 [ 1034.593732] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1035.182463] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1038.690036] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1043.935988] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1043.936137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.936702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1043.943527] Lustre: Skipped 1 previous similar message [ 1048.782779] Lustre: server umount lustre-OST0000 complete [ 1054.176162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1054.176404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1054.180263] Lustre: Skipped 1 previous similar message [ 1054.181922] Lustre: Skipped 1 previous similar message [ 1056.125655] Lustre: server umount lustre-MDT0000 complete [ 1057.321539] LustreError: 79164:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761122273 with bad export cookie 9188118700150770715 [ 1057.322964] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1057.324902] LustreError: 79164:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1057.453543] Lustre: server umount lustre-MDT0001 complete [ 1061.857060] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1062.759883] Key type lgssc unregistered [ 1062.872317] LNet: 82832:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.874641] LNet: Removed LNI 192.168.204.133@tcp [ 1063.153164] Key type .llcrypt unregistered [ 1063.155481] Key type ._llcrypt unregistered [ 1071.452250] alg: No test for adler32 (adler32-zlib) [ 1072.203345] Key type ._llcrypt registered [ 1072.204430] Key type .llcrypt registered [ 1072.240192] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1072.664935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1072.787654] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 1072.861102] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1072.862961] LNet: Accept secure, port 988 [ 1074.439168] Key type lgssc registered [ 1074.771598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1077.306754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1094.815303] LustreError: 84267:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.133@tcp: failed processing log, type 1: rc = -5 [ 1125.535246] LustreError: 84267:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.133@tcp: failed processing log, type 4: rc = -110 [ 1156.319255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.664644] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1166.012473] hrtimer: interrupt took 4069137 ns [ 1169.732376] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1178.695424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.839911] Lustre: *** cfs_fail_loc=706, val=0*** [ 1202.485359] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.542900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.596173] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1205.159746] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1207.777499] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.101732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.300358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1210.337161] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1212.200348] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1216.451754] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1219.502150] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1220.576299] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1220.581526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.590664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1223.137217] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1223.141208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.146587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1224.675834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1224.682838] Lustre: Skipped 1 previous similar message [ 1226.526507] Lustre: server umount lustre-OST0000 complete [ 1229.793285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.795924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1229.799030] Lustre: Skipped 1 previous similar message [ 1229.803614] Lustre: Skipped 1 previous similar message [ 1234.914556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.917044] Lustre: Skipped 1 previous similar message [ 1243.103161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1243.200309] Lustre: server umount lustre-MDT0000 complete [ 1245.152518] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1245.164385] LustreError: Skipped 1 previous similar message [ 1245.519086] LustreError: 84274:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761122462 with bad export cookie 1621196433551813157 [ 1245.524410] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.528590] LustreError: 84274:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1245.685106] Lustre: server umount lustre-MDT0001 complete [ 1251.665780] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1253.283596] Key type lgssc unregistered [ 1253.444377] LNet: 88015:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.449432] LNet: Removed LNI 192.168.204.133@tcp [ 1253.933278] Key type .llcrypt unregistered [ 1253.934599] Key type ._llcrypt unregistered [ 1265.433434] alg: No test for adler32 (adler32-zlib) [ 1266.185374] Key type ._llcrypt registered [ 1266.187143] Key type .llcrypt registered [ 1266.232520] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1266.704971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1266.881617] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 1266.987140] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1266.989569] LNet: Accept secure, port 988 [ 1268.575129] Key type lgssc registered [ 1269.101938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1272.652289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.633142] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1279.061931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.315752] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1279.361334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1281.297827] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1284.573303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.576400] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.587912] LustreError: Skipped 2 previous similar messages [ 1284.705553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.494709] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1290.058944] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.064771] LustreError: Skipped 3 previous similar messages [ 1290.160656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.186235] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 1292.183480] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1295.282840] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 1305.568977] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1305.571181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.572015] LustreError: Skipped 1 previous similar message [ 1305.576696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1305.579975] Lustre: Skipped 1 previous similar message [ 1307.312912] Lustre: server umount lustre-OST0000 complete [ 1310.690821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.691267] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1310.694314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1310.695460] Lustre: Skipped 2 previous similar messages [ 1315.355364] Lustre: server umount lustre-MDT0000 complete [ 1315.808027] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.814156] LustreError: Skipped 1 previous similar message [ 1317.246407] Lustre: server umount lustre-MDT0001 complete [ 1324.511222] Lustre: 92880:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761122535/real 1761122535] req@00000000cde447fd x1846670768745600/t0(0) o251->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761122541 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1324.531760] Lustre: server umount MGS complete [ 1330.489489] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:42:26 (1761122546) [ 1344.394585] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 1348.378370] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1353.231933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1356.850641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1360.019852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1363.191634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1366.353104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1366.382097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.351512] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1373.335760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1373.372303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.476567] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1373.498862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1373.542522] Lustre: lustre-MDT0001: new disk, initializing [ 1373.583986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1373.596710] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1373.600272] LustreError: 97578:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 1375.606689] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1381.546393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1381.580796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1381.585249] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1381.720861] Lustre: lustre-OST0000: new disk, initializing [ 1381.723938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1381.759260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.853283] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1385.249398] Lustre: 98734:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000f43e1159 x1846670768750080/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761122613 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1389.019549] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1392.429393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1392.454340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1392.499932] Lustre: lustre-MDT0000: new disk, initializing [ 1392.561299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1392.572608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1393.632288] Lustre: 99428:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@000000006e0383d6 x1846670768753344/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1761122622 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1394.563349] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1395.681319] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1398.756931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1398.763154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1399.783270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1399.789078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1420.256444] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1420.260956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.273549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.276532] Lustre: Skipped 1 previous similar message [ 1423.330711] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1423.330839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1423.334664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.612210] Lustre: server umount lustre-OST0000 complete [ 1428.449457] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1428.451086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.463806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1428.468222] Lustre: Skipped 1 previous similar message [ 1428.474056] Lustre: Skipped 1 previous similar message [ 1433.808500] Lustre: server umount lustre-MDT0000 complete [ 1449.951203] Lustre: 100620:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761122660/real 1761122660] req@000000008da9bfba x1846670768762880/t0(0) o251->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761122666 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1449.973810] Lustre: server umount MGS complete [ 1449.975398] Lustre: Skipped 1 previous similar message [ 1455.244106] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:44:31 (1761122671) [ 1465.525027] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1467.505749] Key type lgssc unregistered [ 1467.688632] LNet: 102026:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.692844] LNet: Removed LNI 192.168.204.133@tcp [ 1468.267662] Key type .llcrypt unregistered [ 1468.269362] Key type ._llcrypt unregistered [ 1479.351466] alg: No test for adler32 (adler32-zlib) [ 1480.103465] Key type ._llcrypt registered [ 1480.105491] Key type .llcrypt registered [ 1480.183453] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1480.807623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1481.102632] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 1481.262044] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1481.266114] LNet: Accept secure, port 988 [ 1482.903193] Key type lgssc registered [ 1483.605752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1489.998128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1491.271054] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1491.303096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.756388] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1496.550044] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.103417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.384598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1503.954316] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1509.160632] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1512.967611] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1517.796978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1518.005759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1520.706309] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1522.091367] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 1525.473358] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1542.625493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1542.626834] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1542.627316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1542.630900] Lustre: Skipped 1 previous similar message [ 1546.089698] Lustre: server umount lustre-OST0000 complete [ 1552.864299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1552.864421] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1552.864897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1552.864904] Lustre: Skipped 1 previous similar message [ 1552.869655] Lustre: Skipped 1 previous similar message [ 1552.873642] LustreError: Skipped 1 previous similar message [ 1554.397670] Lustre: server umount lustre-MDT0000 complete [ 1556.151392] LustreError: 103272:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761122772 with bad export cookie 16031276075234524237 [ 1556.153299] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1556.156790] LustreError: 103272:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1556.308735] Lustre: server umount lustre-MDT0001 complete [ 1562.476949] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1563.803597] Key type lgssc unregistered [ 1563.951351] LNet: 106965:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.956594] LNet: Removed LNI 192.168.204.133@tcp [ 1564.333542] Key type .llcrypt unregistered [ 1564.335069] Key type ._llcrypt unregistered [ 1571.946511] alg: No test for adler32 (adler32-zlib) [ 1572.698391] Key type ._llcrypt registered [ 1572.700721] Key type .llcrypt registered [ 1572.754343] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1573.263609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1573.418195] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 1573.519172] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1573.521798] LNet: Accept secure, port 988 [ 1575.111757] Key type lgssc registered [ 1575.571256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1578.683745] LustreError: 108094:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1578.684461] LustreError: 108093:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1578.690484] LustreError: 108094:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4997 [ 1578.693660] LustreError: 108094:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1578.698653] LustreError: 108094:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1578.707201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.899762] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.924184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1583.460210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.613731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1585.605746] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1587.862503] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing lsmod [ 1589.094024] LustreError: 109402:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1589.097316] LustreError: 109403:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1589.099974] LustreError: 109402:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4998 [ 1589.103116] LustreError: 109402:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1589.109049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1589.110784] LustreError: 109402:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1589.297940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1593.825291] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1593.828670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1593.834704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1594.337039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1594.338188] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1594.339569] Lustre: Skipped 1 previous similar message [ 1594.344078] Lustre: Skipped 1 previous similar message [ 1596.749790] Lustre: server umount lustre-MDT0000 complete [ 1598.349946] LustreError: 108111:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761122814 with bad export cookie 17640807100856054657 [ 1598.352376] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1598.354539] LustreError: 108111:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1598.473885] Lustre: server umount lustre-MDT0001 complete [ 1606.111331] Lustre: 109994:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761122816/real 1761122816] req@00000000a35ab3dd x1846671090655040/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761122822 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1606.199550] Lustre: server umount lustre-OST0000 complete [ 1609.673532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.896440] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1609.931377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1611.662363] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1615.254304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.328493] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1615.444507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1617.254874] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1620.237898] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1622.885758] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1625.633648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1625.827963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.535891] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1631.020305] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1640.172155] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 1645.538277] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1645.540685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.545992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1645.548506] Lustre: Skipped 2 previous similar messages [ 1649.753825] Lustre: server umount lustre-OST0000 complete [ 1655.779024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.780073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1655.783604] Lustre: Skipped 2 previous similar messages [ 1655.788364] Lustre: Skipped 1 previous similar message [ 1657.622638] Lustre: server umount lustre-MDT0000 complete [ 1659.277967] LustreError: 110449:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761122875 with bad export cookie 17640807100856055322 [ 1659.281150] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1659.283364] LustreError: 110449:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1665.290561] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1666.876480] Key type lgssc unregistered [ 1667.003585] LNet: 114142:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.008038] LNet: Removed LNI 192.168.204.133@tcp [ 1667.389563] Key type .llcrypt unregistered [ 1667.391294] Key type ._llcrypt unregistered [ 1678.730334] alg: No test for adler32 (adler32-zlib) [ 1679.481364] Key type ._llcrypt registered [ 1679.483114] Key type .llcrypt registered [ 1679.527357] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1679.998411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.186587] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 1680.296431] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1680.298883] LNet: Accept secure, port 988 [ 1681.903213] Key type lgssc registered [ 1682.356350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1685.564264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.739435] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.758155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.509578] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1692.128602] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1692.556487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.750530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1694.692710] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1698.140925] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1701.280601] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1704.311778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1704.495450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.454441] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1709.753522] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1716.777208] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1722.403880] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1723.476139] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1725.924553] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1725.929568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.940238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1726.948027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1726.949451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.790976] Lustre: server umount lustre-OST0000 complete [ 1733.601216] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1733.605271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.612918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1733.615713] Lustre: Skipped 1 previous similar message [ 1737.184332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1737.184903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.649371] Lustre: server umount lustre-MDT0000 complete [ 1740.326972] LustreError: 115584:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761122956 with bad export cookie 14360776681303029032 [ 1740.327982] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.332802] LustreError: 115584:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1740.484920] Lustre: server umount lustre-MDT0001 complete [ 1746.289060] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1747.839848] Key type lgssc unregistered [ 1748.006610] LNet: 119381:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.011286] LNet: Removed LNI 192.168.204.133@tcp [ 1748.411446] Key type .llcrypt unregistered [ 1748.413149] Key type ._llcrypt unregistered [ 1756.628239] alg: No test for adler32 (adler32-zlib) [ 1757.380361] Key type ._llcrypt registered [ 1757.381682] Key type .llcrypt registered [ 1757.431749] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1757.810103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1758.058162] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 1758.155912] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1758.158335] LNet: Accept secure, port 988 [ 1759.743856] Key type lgssc registered [ 1760.300199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1763.992595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.207070] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1765.233906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1767.124529] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1770.464197] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1771.119810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.290825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1773.271678] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1776.979408] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1779.625681] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1783.246645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1783.408063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.408614] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1789.133125] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1794.669506] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1804.773877] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1804.780409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.792048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1805.284510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1805.294064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1805.297533] Lustre: Skipped 1 previous similar message [ 1809.482554] Lustre: server umount lustre-OST0000 complete [ 1812.448045] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1812.456495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1812.463328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1815.523395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.524165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1815.534225] Lustre: Skipped 1 previous similar message [ 1817.841892] Lustre: server umount lustre-MDT0000 complete [ 1820.147937] LustreError: 120628:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761123036 with bad export cookie 9679758774487885584 [ 1820.154498] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1820.323270] Lustre: server umount lustre-MDT0001 complete [ 1828.054830] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1830.085525] Key type lgssc unregistered [ 1830.334690] LNet: 124322:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.342623] LNet: Removed LNI 192.168.204.133@tcp [ 1830.955454] Key type .llcrypt unregistered [ 1830.958730] Key type ._llcrypt unregistered [ 1848.683032] alg: No test for adler32 (adler32-zlib) [ 1849.434602] Key type ._llcrypt registered [ 1849.435895] Key type .llcrypt registered [ 1849.503051] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1850.082931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.261245] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 1850.361449] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1850.364403] LNet: Accept secure, port 988 [ 1851.975168] Key type lgssc registered [ 1852.542446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1856.856668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1858.138378] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.180079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.662844] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1863.649492] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1865.572384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.785986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1868.225901] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1872.330079] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1875.585886] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1878.994790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1879.177644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.409866] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1885.573251] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1889.453082] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1894.298751] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1896.634931] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1896.639902] Lustre: Skipped 2 previous similar messages [ 1899.329702] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1899.335481] Lustre: Skipped 2 previous similar messages [ 1902.613456] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1902.630973] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1905.754970] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1905.758627] Lustre: Skipped 2 previous similar messages [ 1910.813119] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.33@tcp 192.168.0.[2,10]@tcp [ 1913.314342] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1913.323624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1913.332947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1914.994475] Lustre: server umount lustre-OST0000 complete [ 1920.481388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.483867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1920.486342] Lustre: Skipped 1 previous similar message [ 1922.527434] Lustre: 129682:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761123133/real 1761123133] req@00000000fbd41054 x1846671380075712/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761123139 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1924.711212] Lustre: server umount lustre-MDT0000 complete [ 1925.601142] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1925.612211] LustreError: Skipped 1 previous similar message [ 1926.768810] LustreError: 125765:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761123143 with bad export cookie 2429933130230523503 [ 1926.770151] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1926.776169] LustreError: 125765:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1926.969523] Lustre: server umount lustre-MDT0001 complete [ 1933.127429] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1934.293517] Key type lgssc unregistered [ 1934.424601] LNet: 130486:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1934.428901] LNet: Removed LNI 192.168.204.133@tcp [ 1934.793455] Key type .llcrypt unregistered [ 1934.794811] Key type ._llcrypt unregistered [ 1949.545660] alg: No test for adler32 (adler32-zlib) [ 1950.297385] Key type ._llcrypt registered [ 1950.298778] Key type .llcrypt registered [ 1950.348231] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1950.927882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1951.089050] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 1951.199816] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1951.202608] LNet: Accept secure, port 988 [ 1952.823125] Key type lgssc registered [ 1953.473178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1958.372811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.581861] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.607407] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1959.618901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.621780] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.204.33@tcp 192.168.0.[2,10]@tcp [ 1961.574812] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1965.024394] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.231365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.379569] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1966.391029] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.33@tcp 192.168.0.[2,10]@tcp [ 1966.414307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1968.414358] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1972.495412] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1975.062137] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1978.251563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1978.415271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.346522] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 1984.400274] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1989.672912] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 1994.720621] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1994.722306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.724951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1994.729181] Lustre: Skipped 1 previous similar message [ 1994.733594] Lustre: Skipped 1 previous similar message [ 1996.009291] Lustre: server umount lustre-OST0000 complete [ 1999.841710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.843358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1999.846382] Lustre: Skipped 1 previous similar message [ 2004.142672] Lustre: server umount lustre-MDT0000 complete [ 2004.961192] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.969973] LustreError: Skipped 1 previous similar message [ 2006.381140] LustreError: 132072:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761123222 with bad export cookie 15356979927051579913 [ 2006.384251] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2006.386354] LustreError: 132072:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2006.549062] Lustre: server umount lustre-MDT0001 complete [ 2013.009860] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 2014.453837] Key type lgssc unregistered [ 2014.592905] LNet: 135797:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2014.597963] LNet: Removed LNI 192.168.204.133@tcp [ 2014.981755] Key type .llcrypt unregistered [ 2014.985169] Key type ._llcrypt unregistered [ 2029.390993] alg: No test for adler32 (adler32-zlib) [ 2030.145475] Key type ._llcrypt registered [ 2030.146893] Key type .llcrypt registered [ 2030.221540] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 2038.201687] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2038.888166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2039.096095] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 2039.216608] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 2039.219165] LNet: Accept secure, port 988 [ 2040.816099] Key type lgssc registered [ 2041.453372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2046.539183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2050.627559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2053.826243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2056.789581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2059.988091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2060.036787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.160418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2061.177308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2061.220667] Lustre: lustre-MDT0000: new disk, initializing [ 2061.262333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.275070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2063.101945] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2068.858593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2068.904198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.954538] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2068.981779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2068.985515] Lustre: Skipped 1 previous similar message [ 2069.027907] Lustre: lustre-MDT0001: new disk, initializing [ 2069.082382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2069.104129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2069.110797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2071.501360] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2076.871725] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2079.699066] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2082.689321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2082.735415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2082.739790] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2082.864876] Lustre: lustre-OST0000: new disk, initializing [ 2082.867684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2082.907702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.927251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2083.931023] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2085.005155] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2090.224021] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2092.446335] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2092.548036] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2094.598640] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2094.694951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2099.681317] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2099.685037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.692730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2099.695445] Lustre: Skipped 1 previous similar message [ 2101.737672] Lustre: server umount lustre-OST0000 complete [ 2104.800943] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2104.802470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.804841] LustreError: Skipped 1 previous similar message [ 2104.808371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2104.810713] Lustre: Skipped 2 previous similar messages [ 2109.716378] Lustre: server umount lustre-MDT0000 complete [ 2109.920085] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.926192] LustreError: Skipped 1 previous similar message [ 2111.397225] LustreError: 140166:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761123327 with bad export cookie 15446517001714183413 [ 2111.401079] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.402744] LustreError: 140166:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2111.552825] Lustre: server umount lustre-MDT0001 complete [ 2115.569143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.804133] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.838310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.013669] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2121.185957] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.697709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.916557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2125.147330] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2128.659599] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2131.296515] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2134.540435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2134.722377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.623483] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2140.133744] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2142.250482] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2146.504338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2148.772373] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2148.879578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2153.748860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2153.781127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2153.785678] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2153.857128] Lustre: lustre-OST0001: new disk, initializing [ 2153.860144] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2153.911934] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2156.444579] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2161.142347] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2161.194956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2161.201281] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2162.346679] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2164.521735] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 2164.635492] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2167.471349] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9dc148d69000.ost_server_uuid 40 [ 2168.186517] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9dc148d69000.ost_server_uuid in FULL state after 0 sec [ 2176.484234] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2176.486498] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.487902] LustreError: Skipped 1 previous similar message [ 2176.493735] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2176.498300] Lustre: Skipped 2 previous similar messages [ 2177.478859] Lustre: server umount lustre-OST0001 complete [ 2181.604191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.604598] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.608676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2181.610918] Lustre: Skipped 2 previous similar messages [ 2181.616803] LustreError: Skipped 1 previous similar message [ 2185.389253] Lustre: server umount lustre-OST0000 complete [ 2189.792518] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2189.796170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.802123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.806205] Lustre: Skipped 1 previous similar message [ 2193.566795] Lustre: server umount lustre-MDT0000 complete [ 2195.334487] LustreError: 143650:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761123411 with bad export cookie 15446517001714184267 [ 2195.337129] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.340121] LustreError: 143650:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2205.890102] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 2207.365167] Key type lgssc unregistered [ 2207.520542] LNet: 150349:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.526354] LNet: Removed LNI 192.168.204.133@tcp [ 2207.919862] Key type .llcrypt unregistered [ 2207.922114] Key type ._llcrypt unregistered [ 2225.638842] alg: No test for adler32 (adler32-zlib) [ 2226.391475] Key type ._llcrypt registered [ 2226.392755] Key type .llcrypt registered [ 2226.436216] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 2233.536510] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2234.011408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2234.241181] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 2234.344535] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 2234.348188] LNet: Accept secure, port 988 [ 2235.943180] Key type lgssc registered [ 2236.424583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2241.064313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2245.206988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2248.861531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2252.683631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2256.696866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2256.738462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2257.888074] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2257.912461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2257.958220] Lustre: lustre-MDT0000: new disk, initializing [ 2258.006147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.018225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2260.233791] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2266.147202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2266.186908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.245241] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2266.280826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2266.286884] Lustre: Skipped 1 previous similar message [ 2266.332295] Lustre: lustre-MDT0001: new disk, initializing [ 2266.383312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2266.405190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2266.410925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2268.565531] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2274.320344] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2277.382229] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2281.262052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2281.307439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2281.311974] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2281.452317] Lustre: lustre-OST0000: new disk, initializing [ 2281.455452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2281.501124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.485818] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2288.947170] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2289.180287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2289.187216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2291.225470] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2291.325780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2293.356586] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2293.463643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2299.360950] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2299.362954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.366815] LustreError: Skipped 1 previous similar message [ 2299.368789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.372145] Lustre: Skipped 1 previous similar message [ 2300.523341] Lustre: server umount lustre-OST0000 complete [ 2304.481974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2304.482766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.488307] Lustre: Skipped 1 previous similar message [ 2304.491562] Lustre: Skipped 1 previous similar message [ 2308.760631] Lustre: server umount lustre-MDT0000 complete [ 2309.601154] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.610479] LustreError: Skipped 1 previous similar message [ 2310.989059] LustreError: 158117:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761123527 with bad export cookie 17903167867244521014 [ 2310.995526] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2310.995778] LustreError: 158117:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2311.189974] Lustre: server umount lustre-MDT0001 complete [ 2315.347649] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 04:58:51 (1761123531) [ 2323.950264] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 2328.007137] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2332.404914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2335.318268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2338.118189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2341.281601] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2344.722579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2344.754960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.880827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2344.902055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2344.947924] Lustre: lustre-MDT0000: new disk, initializing [ 2344.989852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2345.003409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2347.102359] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2352.600640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2352.644299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2352.696214] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2352.722528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2352.725578] Lustre: Skipped 1 previous similar message [ 2352.770907] Lustre: lustre-MDT0001: new disk, initializing [ 2352.822606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2352.841804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2352.848701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2354.807246] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2360.408629] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2363.463125] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2366.908543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2366.941114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2366.946816] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2367.067637] Lustre: lustre-OST0000: new disk, initializing [ 2367.070615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2367.109928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.006491] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2374.063681] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2376.203425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2376.209780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2381.861942] Lustre: Failing over lustre-OST0000 [ 2381.949107] Lustre: server umount lustre-OST0000 complete [ 2392.543253] Lustre: 152664:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761123602/real 1761123602] req@000000009272a8d7 x1846671782737728/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761123609 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2392.543389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.554355] Lustre: 152664:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2396.376628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.532566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2396.540479] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2398.480492] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2398.559727] Lustre: 152664:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761123608/real 1761123608] req@00000000da9b5393 x1846671782738432/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761123615 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2398.565968] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2398.568882] Lustre: 152664:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2399.066161] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2399.066190] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2399.069847] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2400.164732] Lustre: Failing over lustre-MDT0000 [ 2400.305275] Lustre: server umount lustre-MDT0000 complete [ 2401.183286] Lustre: 152663:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761123609/real 1761123609] req@0000000076ca71d5 x1846671782738880/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761123616 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2403.809925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2403.811575] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.814751] Lustre: Skipped 3 previous similar messages [ 2403.820741] LustreError: Skipped 1 previous similar message [ 2408.928446] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.937342] LustreError: Skipped 3 previous similar messages [ 2410.975564] Lustre: 152665:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761123620/real 1761123620] req@00000000c1f74939 x1846671782742976/t0(0) o400->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761123627 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2410.990665] Lustre: 152665:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2410.995495] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2411.008490] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.014912] LustreError: Skipped 3 previous similar messages [ 2415.311685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.442588] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.457083] LustreError: Skipped 1 previous similar message [ 2417.127615] Lustre: Evicted from MGS (at 192.168.204.133@tcp) after server handle changed from 0xf874d451f94c298c to 0xf874d451f94c2f27 [ 2417.135448] Lustre: MGC192.168.204.133@tcp: Connection restored to (at 0@lo) [ 2417.138282] Lustre: Skipped 1 previous similar message [ 2417.280726] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2419.420601] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2420.561393] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2422.758438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2422.767461] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2422.789633] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2424.800180] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2424.803736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.809263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.812036] Lustre: Skipped 1 previous similar message [ 2427.876316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2427.877377] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.379944] Lustre: server umount lustre-OST0000 complete [ 2432.992273] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2432.992860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.996633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2433.001317] Lustre: Skipped 2 previous similar messages [ 2433.006195] Lustre: Skipped 2 previous similar messages [ 2440.056654] LustreError: 162268:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761123656 with bad export cookie 17903167867244523303 [ 2440.061033] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2440.061429] LustreError: 162268:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2446.503060] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 2447.843925] Key type lgssc unregistered [ 2447.981935] LNet: 167282:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.985829] LNet: Removed LNI 192.168.204.133@tcp [ 2448.390384] Key type .llcrypt unregistered [ 2448.392097] Key type ._llcrypt unregistered [ 2459.899347] alg: No test for adler32 (adler32-zlib) [ 2460.652470] Key type ._llcrypt registered [ 2460.654157] Key type .llcrypt registered [ 2460.711933] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2461.216468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2461.432610] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 2461.546435] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 2461.548912] LNet: Accept secure, port 988 [ 2463.160822] Key type lgssc registered [ 2463.755751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2467.861240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.071827] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.097622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.009154] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2474.464342] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.698302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.864931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2476.823445] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2480.166685] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2482.741445] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2485.801016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2485.953578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.777455] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 2491.478480] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2499.301907] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 2515.809133] Lustre: lustre-MDT0000: Client 64379462-819b-4d8c-9f13-10d08919b05f (at 192.168.204.33@tcp) reconnecting [ 2579.348610] Lustre: lustre-MDT0000: Client 64379462-819b-4d8c-9f13-10d08919b05f (at 192.168.204.33@tcp) reconnecting [ 2676.609076] Lustre: lustre-MDT0000: Client 64379462-819b-4d8c-9f13-10d08919b05f (at 192.168.204.33@tcp) reconnecting [ 2725.767639] Lustre: lustre-MDT0000: Client 64379462-819b-4d8c-9f13-10d08919b05f (at 192.168.204.33@tcp) reconnecting [ 2770.793097] Lustre: lustre-MDT0000: Client 64379462-819b-4d8c-9f13-10d08919b05f (at 192.168.204.33@tcp) reconnecting [ 2820.015311] Lustre: lustre-MDT0000: Client 64379462-819b-4d8c-9f13-10d08919b05f (at 192.168.204.33@tcp) reconnecting [ 2889.580133] Lustre: lustre-MDT0000: Client 64379462-819b-4d8c-9f13-10d08919b05f (at 192.168.204.33@tcp) reconnecting [ 2951.021813] Lustre: lustre-MDT0000: Client 64379462-819b-4d8c-9f13-10d08919b05f (at 192.168.204.33@tcp) reconnecting [ 3029.863328] Lustre: lustre-MDT0000: Client 64379462-819b-4d8c-9f13-10d08919b05f (at 192.168.204.33@tcp) reconnecting [ 3089.260632] Lustre: Failing over lustre-MDT0000 [ 3089.374493] Lustre: server umount lustre-MDT0000 complete [ 3089.382353] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3089.387306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.401078] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3093.337188] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3093.480096] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.488555] Lustre: Skipped 1 previous similar message [ 3098.457872] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.477356] LustreError: Skipped 3 previous similar messages [ 3099.615190] Lustre: 168054:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761124309/real 1761124309] req@00000000969b89c4 x1846672022174336/t0(0) o400->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761124316 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3099.631611] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3100.517246] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.532389] LustreError: Skipped 6 previous similar messages [ 3104.739630] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.746089] LustreError: Skipped 2 previous similar messages [ 3105.966148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.001045] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.007831] LustreError: Skipped 11 previous similar messages [ 3117.041191] Lustre: Evicted from MGS (at 192.168.204.133@tcp) after server handle changed from 0x4ad5a4b382331c0e to 0x4ad5a4b38235884b [ 3117.054830] Lustre: MGC192.168.204.133@tcp: Connection restored to 192.168.204.133@tcp (at 0@lo) [ 3117.215231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.263173] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3120.125033] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3122.006300] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3122.663168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3122.669262] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3122.706688] Lustre: lustre-OST0000: deleting orphan objects from 0x0:100 to 0x0:129 [ 3126.480842] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3127.728337] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3143.136642] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3143.140319] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.149382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3143.153141] Lustre: Skipped 1 previous similar message [ 3148.258974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3148.263845] Lustre: Skipped 4 previous similar messages [ 3148.346785] Lustre: server umount lustre-MDT0000 complete [ 3150.519459] LustreError: 168723:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761124367 with bad export cookie 5392397219736684619 [ 3150.523576] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3150.526710] LustreError: 168723:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3153.379487] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.379845] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.391019] LustreError: Skipped 4 previous similar messages [ 3153.397364] Lustre: Skipped 1 previous similar message [ 3153.398916] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3155.424818] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3156.849368] Lustre: server umount lustre-MDT0001 complete [ 3173.855362] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3173.956055] Lustre: server umount lustre-OST0000 complete [ 3178.915676] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 3183.499375] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3187.944497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3190.917835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3193.784739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3197.236577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3200.797069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3200.851663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3201.013723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3201.033981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3201.079894] Lustre: lustre-MDT0000: new disk, initializing [ 3201.140271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3201.162503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3203.518231] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3209.314898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3209.361383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.442349] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3209.465383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3209.468167] Lustre: Skipped 1 previous similar message [ 3209.510938] Lustre: lustre-MDT0001: new disk, initializing [ 3209.560844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3209.584485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3209.591861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3211.709168] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3217.297439] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3220.405213] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3223.955303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3224.000857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.005044] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3224.132809] Lustre: lustre-OST0000: new disk, initializing [ 3224.137215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3224.180776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3226.355641] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3231.773188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3231.779101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3232.325394] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3234.670238] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3234.765432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3236.944928] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3237.047542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3241.957199] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3241.958450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3241.965378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3241.968182] Lustre: Skipped 1 previous similar message [ 3244.235816] Lustre: server umount lustre-OST0000 complete [ 3252.192778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.196389] Lustre: Skipped 4 previous similar messages [ 3252.508669] Lustre: server umount lustre-MDT0000 complete [ 3254.528782] LustreError: 175696:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761124471 with bad export cookie 5392397219736685312 [ 3254.532221] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3254.538213] LustreError: 175696:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3259.213963] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:14:35 (1761124475) [ 3267.417523] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 3271.778975] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3276.302806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3279.314745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3282.262754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3285.410134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3289.292472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3289.342768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3289.478350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3289.505756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3289.550332] Lustre: lustre-MDT0000: new disk, initializing [ 3289.614189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.632167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3291.783391] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3297.529796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3297.568939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.627781] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3297.654506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3297.657832] Lustre: Skipped 1 previous similar message [ 3297.703968] Lustre: lustre-MDT0001: new disk, initializing [ 3297.768118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3297.790715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3297.795625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3300.025948] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3306.195312] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3309.670231] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3313.689340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3313.740414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3313.744659] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3313.871350] Lustre: lustre-OST0000: new disk, initializing [ 3313.874465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3313.921300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.090692] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3319.821526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3319.825906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3321.530510] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3324.896901] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3324.898395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.901178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3324.905720] Lustre: Skipped 3 previous similar messages [ 3339.231128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3339.324664] Lustre: server umount lustre-OST0000 complete [ 3339.326833] Lustre: Skipped 1 previous similar message [ 3343.840587] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3343.845238] LustreError: Skipped 1 previous similar message [ 3343.848461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3350.154144] LustreError: 182676:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761124566 with bad export cookie 5392397219736686159 [ 3350.155871] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3350.165870] LustreError: 182676:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3355.504323] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:16:11 (1761124571) [ 3364.831405] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 3369.882939] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3375.304359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3378.812734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3381.830777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3384.847611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3388.498825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3388.535778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.677772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3388.704130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3388.753847] Lustre: lustre-MDT0000: new disk, initializing [ 3388.831555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.848769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3390.970586] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3396.615783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3396.653228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3396.712355] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3396.844140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3398.970205] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3404.444606] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3407.648820] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3411.489357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3411.535820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3411.540089] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3414.275405] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3418.125790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3419.928462] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3425.766534] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3425.771053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.778549] Lustre: Skipped 1 previous similar message [ 3425.783047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3425.785684] Lustre: Skipped 8 previous similar messages [ 3429.775310] Lustre: server umount lustre-OST0000 complete [ 3429.777509] Lustre: Skipped 2 previous similar messages [ 3438.560347] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.571601] LustreError: Skipped 2 previous similar messages [ 3440.235625] LustreError: 189134:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761124656 with bad export cookie 5392397219736687244 [ 3440.240637] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3440.242407] LustreError: 189134:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3447.280767] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 3448.822682] Key type lgssc unregistered [ 3448.993789] LNet: 193096:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3448.999173] LNet: Removed LNI 192.168.204.133@tcp [ 3449.465785] Key type .llcrypt unregistered [ 3449.467442] Key type ._llcrypt unregistered [ 3462.293448] alg: No test for adler32 (adler32-zlib) [ 3463.045518] Key type ._llcrypt registered [ 3463.047322] Key type .llcrypt registered [ 3463.093792] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3463.564565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3463.740659] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 3463.842729] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3463.845661] LNet: Accept secure, port 988 [ 3465.455100] Key type lgssc registered [ 3465.981314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3469.576884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.823910] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.855252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.300448] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3475.938588] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3477.975654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.179383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3480.263695] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3484.019670] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3486.547237] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3489.580928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3489.747771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3491.728067] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3495.487830] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3501.034121] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 3518.433556] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3518.438324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.446201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3521.505414] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3521.505947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3521.508725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.137249] Lustre: server umount lustre-OST0000 complete [ 3526.625437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.631672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3526.634295] Lustre: Skipped 2 previous similar messages [ 3531.744476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3531.748111] Lustre: Skipped 1 previous similar message [ 3531.923351] Lustre: server umount lustre-MDT0000 complete [ 3533.503042] LustreError: 194536:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761124750 with bad export cookie 4865475315692977025 [ 3533.506053] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3533.508750] LustreError: 194536:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3533.650154] Lustre: server umount lustre-MDT0001 complete [ 3539.183045] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 3540.371345] Key type lgssc unregistered [ 3540.506620] LNet: 198231:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3540.510913] LNet: Removed LNI 192.168.204.133@tcp [ 3540.846075] Key type .llcrypt unregistered [ 3540.847748] Key type ._llcrypt unregistered [ 3551.425705] alg: No test for adler32 (adler32-zlib) [ 3552.177397] Key type ._llcrypt registered [ 3552.178623] Key type .llcrypt registered [ 3552.224484] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3552.701640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3552.878518] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 3552.980982] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3552.984812] LNet: Accept secure, port 988 [ 3554.599152] Key type lgssc registered [ 3555.098900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3558.810111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3560.034552] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.064605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.925212] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3565.536388] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3565.979133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.165108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3568.086652] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3571.466918] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3573.783248] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3576.545636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3576.694865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3578.484544] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3581.737340] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3591.014376] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 3596.256088] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3596.259265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.261843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.265649] Lustre: Skipped 1 previous similar message [ 3597.665508] Lustre: lustre-OST0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 3597.668537] Lustre: Skipped 1 previous similar message [ 3601.326414] Lustre: server umount lustre-OST0000 complete [ 3604.784915] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9dc1506d6000.ost_server_uuid 40 [ 3605.521309] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3616.737454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3616.743360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3616.745936] Lustre: Skipped 1 previous similar message [ 3621.860859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3621.864509] Lustre: Skipped 1 previous similar message [ 3622.122994] Lustre: server umount lustre-MDT0000 complete [ 3624.095185] LustreError: 201644:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761124840 with bad export cookie 2382410195030126986 [ 3624.098448] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3624.100309] LustreError: 201644:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3624.248718] Lustre: server umount lustre-MDT0001 complete [ 3628.425049] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:20:44 (1761124844) [ 3631.984986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.254302] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3632.300988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.640669] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3637.731058] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.131211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.314259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3641.367064] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3644.927674] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3647.893816] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3651.216178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3651.372478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.091846] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3656.226246] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3659.124907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3659.170870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3659.174649] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3659.242991] Lustre: lustre-OST0001: new disk, initializing [ 3659.245701] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3661.165856] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3663.891648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3663.895890] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3664.933306] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 3665.792101] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3669.987933] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3669.988098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3669.988610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3669.991302] LustreError: Skipped 2 previous similar messages [ 3670.001396] Lustre: Skipped 2 previous similar messages [ 3680.224945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3680.227717] Lustre: Skipped 6 previous similar messages [ 3681.247222] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3681.270631] Lustre: server umount lustre-OST0000 complete [ 3681.616585] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.941212] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3684.039768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3685.918185] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3686.027333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3690.469593] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3690.476129] LustreError: Skipped 4 previous similar messages [ 3699.681368] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3699.681918] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3699.683980] LustreError: Skipped 1 previous similar message [ 3699.684701] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3699.696596] Lustre: Skipped 1 previous similar message [ 3702.707192] Lustre: server umount lustre-OST0001 complete [ 3705.827306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.827306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.827387] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3710.741765] Lustre: server umount lustre-MDT0000 complete [ 3710.944910] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3710.950829] LustreError: Skipped 6 previous similar messages [ 3712.409393] LustreError: 203659:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761124928 with bad export cookie 2382410195030128036 [ 3712.412066] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3712.414486] LustreError: 203659:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3715.116083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3716.847404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3718.883445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3720.757374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3723.544934] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 05:22:19 (1761124939) [ 3726.378606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3726.528728] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3726.546432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3726.632852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.636192] Lustre: Skipped 1 previous similar message [ 3728.511877] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3732.296875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.354755] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3732.374215] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3734.476820] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3738.071437] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3740.711301] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3743.699237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3743.799739] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3745.823791] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3749.423491] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3752.503183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3752.544107] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3752.676231] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3754.489431] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3757.812376] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3763.169479] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3763.171487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.172833] LustreError: Skipped 1 previous similar message [ 3763.173306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.177912] Lustre: Skipped 1 previous similar message [ 3763.179670] Lustre: Skipped 5 previous similar messages [ 3765.168629] Lustre: server umount lustre-OST0000 complete [ 3765.170620] Lustre: Skipped 1 previous similar message [ 3768.293276] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.300965] LustreError: Skipped 1 previous similar message [ 3778.529324] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3778.533389] LustreError: Skipped 1 previous similar message [ 3788.768550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3788.774358] Lustre: Skipped 3 previous similar messages [ 3791.477504] LustreError: 210074:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761125007 with bad export cookie 2382410195030129422 [ 3791.481150] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3791.481867] LustreError: 210074:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3793.926749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3795.525207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3797.110702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3798.696669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3801.221691] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 05:23:37 (1761125017) [ 3808.044937] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 3811.623979] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3815.712192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3818.090049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3820.273885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3822.902511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3825.842575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3825.879702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3825.982944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3826.001097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3826.044296] Lustre: lustre-MDT0000: new disk, initializing [ 3826.075285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3826.078279] Lustre: Skipped 3 previous similar messages [ 3826.086408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3827.863534] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3832.729261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3832.767111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3832.808093] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3832.830684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3832.833091] Lustre: Skipped 1 previous similar message [ 3832.871749] Lustre: lustre-MDT0001: new disk, initializing [ 3832.939232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3832.943942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3834.831809] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3839.319663] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3841.486420] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3844.173255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3844.204606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3844.208441] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3844.318895] Lustre: lustre-OST0000: new disk, initializing [ 3844.321804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3846.189826] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3850.786027] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3852.649693] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3852.799881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3852.804410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3853.769255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3855.689534] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3855.786834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3857.888476] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3857.894665] LustreError: Skipped 1 previous similar message [ 3857.896504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3857.902202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3857.904830] Lustre: Skipped 7 previous similar messages [ 3862.687515] Lustre: server umount lustre-OST0000 complete [ 3862.689230] Lustre: Skipped 3 previous similar messages [ 3872.009901] LustreError: 219136:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761125088 with bad export cookie 2382410195030130850 [ 3872.014218] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3872.015297] LustreError: 219136:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3875.528522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3875.729118] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3875.736803] LustreError: Skipped 6 previous similar messages [ 3877.498778] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3881.134655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.994956] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3886.076680] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3888.404340] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3891.216749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3891.386437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3891.389750] Lustre: Skipped 4 previous similar messages [ 3893.070136] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3896.109109] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3897.883123] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3906.186555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3907.905363] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3908.004417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3911.137251] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3911.142105] LustreError: Skipped 1 previous similar message [ 3917.638796] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3919.491627] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3944.052368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3946.254546] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3949.465516] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3950.158811] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3957.491292] Lustre: server umount lustre-OST0000 complete [ 3957.493420] Lustre: Skipped 3 previous similar messages [ 3962.848667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3962.853962] Lustre: Skipped 6 previous similar messages [ 3966.770465] LustreError: 222624:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761125183 with bad export cookie 2382410195030131690 [ 3966.773891] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3966.777143] LustreError: 222624:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3970.000281] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 05:26:26 (1761125186) [ 3972.419581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3974.175638] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3977.341654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.171226] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3982.071203] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3984.316979] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3987.104960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3988.886354] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 3991.707788] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3993.367978] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3993.445643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3995.025864] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3995.110705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3997.804275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3997.841876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3997.845979] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3997.904255] Lustre: lustre-OST0001: new disk, initializing [ 3997.906475] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3999.684940] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4004.311147] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4006.237619] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 4007.941404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4007.946138] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4008.396308] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4010.145336] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 4010.244364] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4013.025766] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4013.028791] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4013.029222] LustreError: Skipped 1 previous similar message [ 4013.031340] Lustre: Skipped 8 previous similar messages [ 4018.148819] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4018.154706] LustreError: Skipped 3 previous similar messages [ 4019.717581] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 4019.814837] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4021.543983] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 4021.630835] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4045.523711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4045.613548] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4045.616651] Lustre: Skipped 5 previous similar messages [ 4047.240077] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4050.339605] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4050.919325] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4050.925557] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4050.925863] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 4050.933634] Lustre: Skipped 1 previous similar message [ 4051.108611] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4082.175242] LustreError: 228909:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761125298 with bad export cookie 2382410195030132628 [ 4082.178078] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4082.179051] LustreError: 228909:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4088.393269] Lustre: server umount lustre-MDT0001 complete [ 4088.395224] Lustre: Skipped 6 previous similar messages [ 4090.470253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4091.954746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4093.357096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4094.789228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4097.377565] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 05:28:33 (1761125313) [ 4100.140985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.270406] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4100.282772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4101.969550] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4105.460173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4105.524636] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4107.398105] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4110.649380] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4113.057379] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4115.638526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4115.719821] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4117.494120] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4120.941848] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4128.416958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4128.562943] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4132.949291] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4142.234903] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4144.422517] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4150.436907] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 4150.693530] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4156.145233] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 4156.334548] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4163.944981] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9dc160040800.ost_server_uuid 40 [ 4166.341502] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9dc160040800.ost_server_uuid in IDLE state after 0 sec [ 4167.595595] Lustre: Permanently deactivating lustre-OST0001 [ 4167.625035] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 4167.627922] Lustre: Skipped 1 previous similar message [ 4171.717960] Lustre: Permanently reactivating lustre-OST0001 [ 4177.379884] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4177.388115] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.390595] LustreError: Skipped 6 previous similar messages [ 4177.405126] Lustre: Skipped 8 previous similar messages [ 4210.143161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4213.380188] LustreError: 237797:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761125429 with bad export cookie 2382410195030133888 [ 4213.386909] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4213.402800] LustreError: 237797:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4218.447658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4221.813235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4225.363184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4228.639564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4233.318739] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 05:30:49 (1761125449) [ 4233.938935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4238.206996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.369179] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4238.390230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4238.394110] Lustre: Skipped 5 previous similar messages [ 4240.972443] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4246.814133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4246.876461] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4249.959884] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4255.650178] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4261.028944] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4268.202550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4268.294505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4268.425703] Lustre: Permanently deactivating lustre-OST0000 [ 4272.578445] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4274.717763] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4279.532241] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4284.902968] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4284.947590] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4284.950819] Lustre: Skipped 1 previous similar message [ 4288.054542] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4290.657431] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 4293.397165] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4294.572868] Lustre: Permanently reactivating lustre-OST0000 [ 4294.583593] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 4294.586608] Lustre: Skipped 4 previous similar messages [ 4297.382185] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4297.391793] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4297.405961] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 4297.423885] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4297.441885] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 4305.889804] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4305.893959] Lustre: Skipped 20 previous similar messages [ 4316.649978] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4316.654247] LustreError: Skipped 31 previous similar messages [ 4332.046941] LustreError: 248267:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761125548 with bad export cookie 2382410195030135498 [ 4332.049506] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4332.062261] LustreError: 248267:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4342.586846] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 4344.650817] Key type lgssc unregistered [ 4344.855813] LNet: 249586:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4344.860525] LNet: Removed LNI 192.168.204.133@tcp [ 4345.406894] Key type .llcrypt unregistered [ 4345.413097] Key type ._llcrypt unregistered [ 4359.983740] alg: No test for adler32 (adler32-zlib) [ 4360.735383] Key type ._llcrypt registered [ 4360.736696] Key type .llcrypt registered [ 4360.838734] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4361.838559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4362.051196] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 4362.226571] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 4362.230830] LNet: Accept secure, port 988 [ 4363.887164] Key type lgssc registered [ 4364.863669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4367.915752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4373.281183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4374.678045] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.742479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.815986] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4380.134686] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4385.249573] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4386.769099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4386.916381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4387.131251] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4387.186631] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4387.193148] Lustre: Permanently deactivating lustre-MDT0001 [ 4387.400470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4388.303126] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4388.319718] LustreError: 251135:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4388.336650] LustreError: 251135:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4388.358696] LustreError: 251135:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 4391.763378] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4400.292361] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4406.845942] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4413.626511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4413.872106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4418.057863] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4421.033554] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4421.053494] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:131 to 0x280000400:161 [ 4425.134425] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4426.148047] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4426.170187] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 4426.173359] LustreError: Skipped 1 previous similar message [ 4431.210801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4431.429846] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4435.249608] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4436.469974] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 [ 4436.476476] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:129 [ 4441.232828] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4444.059289] Lustre: Permanently reactivating lustre-MDT0001 [ 4444.075723] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 4444.078444] Lustre: Skipped 3 previous similar messages [ 4451.711411] Lustre: lustre-MDT0001: Client c9729d4f-f910-455b-b809-b2e2fa8b8044 (at 192.168.204.33@tcp) reconnecting [ 4459.189144] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4461.720944] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4468.828650] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 4469.083556] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4473.462977] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 4473.642401] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4475.761025] Lustre: Permanently deactivating lustre-MDT0001 [ 4475.773891] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 4475.777050] Lustre: Skipped 2 previous similar messages [ 4485.192232] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4498.914799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.922266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4498.923209] Lustre: Skipped 3 previous similar messages [ 4498.934665] Lustre: Skipped 3 previous similar messages [ 4501.216480] Lustre: server umount lustre-MDT0000 complete [ 4503.836891] LustreError: 251084:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761125720 with bad export cookie 16138199637748803655 [ 4503.841963] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4503.848936] LustreError: 251084:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4504.051850] Lustre: server umount lustre-MDT0001 complete [ 4513.148624] Lustre: server umount lustre-OST0000 complete [ 4524.789977] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:35:40 (1761125740) [ 4533.397956] Lustre: server umount lustre-OST0001 complete [ 4538.606516] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 4546.255768] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4552.811981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4557.706036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.603545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4566.299864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4571.141464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4571.211548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.356133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4571.359512] Lustre: Skipped 2 previous similar messages [ 4571.398804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4571.474650] Lustre: lustre-MDT0000: new disk, initializing [ 4571.547428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4571.566252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4574.249485] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4582.730681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4582.790974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4582.840271] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4582.868243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4582.871298] Lustre: Skipped 1 previous similar message [ 4582.948326] Lustre: lustre-MDT0001: new disk, initializing [ 4583.026257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4583.054302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4583.060650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4586.407500] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4593.467726] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4597.051601] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4601.684794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4601.744324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4601.748886] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4601.952994] Lustre: lustre-OST0000: new disk, initializing [ 4601.956722] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4602.019471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4605.063643] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4607.539051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4607.554350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4613.509868] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4615.411102] LustreError: 259615:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4623.460603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4623.525837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4623.530807] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4623.634446] Lustre: lustre-OST0001: new disk, initializing [ 4623.645282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4625.488147] LustreError: 259615:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 4625.511481] LustreError: 259615:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4626.949413] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4631.097886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4631.105075] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4634.116340] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4635.494348] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4635.599131] LustreError: 259615:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 4637.596188] Lustre: lustre-OST0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 4641.772061] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4641.773327] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4641.774150] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4641.804884] Lustre: Skipped 1 previous similar message [ 4643.325194] Lustre: server umount lustre-OST0001 complete [ 4646.887785] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4646.911077] LustreError: Skipped 3 previous similar messages [ 4658.661129] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4658.679699] LustreError: Skipped 1 previous similar message [ 4658.684823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4658.700298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4658.703033] Lustre: Skipped 2 previous similar messages [ 4664.371303] Lustre: server umount lustre-OST0000 complete [ 4669.921647] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4669.925245] LustreError: Skipped 1 previous similar message [ 4669.928722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.936712] Lustre: Skipped 1 previous similar message [ 4669.940104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4669.943836] Lustre: Skipped 3 previous similar messages [ 4675.554058] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4675.564931] LustreError: Skipped 7 previous similar messages [ 4676.199624] LustreError: 262768:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761125892 with bad export cookie 16138199637748806434 [ 4676.201258] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4676.203493] LustreError: 262768:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4682.609074] Lustre: server umount lustre-MDT0001 complete [ 4682.611046] Lustre: Skipped 1 previous similar message [ 4692.344490] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 4694.828602] Key type lgssc unregistered [ 4695.102919] LNet: 264802:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4695.113084] LNet: Removed LNI 192.168.204.133@tcp [ 4695.781746] Key type .llcrypt unregistered [ 4695.787368] Key type ._llcrypt unregistered [ 4721.242323] alg: No test for adler32 (adler32-zlib) [ 4721.996417] Key type ._llcrypt registered [ 4721.998534] Key type .llcrypt registered [ 4722.097494] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 4733.650492] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4734.352582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4734.631046] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 4734.838048] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 4734.841973] LNet: Accept secure, port 988 [ 4736.522243] Key type lgssc registered [ 4737.611699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4746.450027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4751.970986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4757.794250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4763.039872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4768.689768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4768.755983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4769.930579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4769.962697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4770.049283] Lustre: lustre-MDT0000: new disk, initializing [ 4770.134449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4770.154855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4773.735996] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4782.584565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4782.632963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4782.693954] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4782.733309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4782.737033] Lustre: Skipped 1 previous similar message [ 4782.821262] Lustre: lustre-MDT0001: new disk, initializing [ 4782.963480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4783.009765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4783.020593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4786.704234] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4794.306541] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4798.852347] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4804.515895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4804.583633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4804.588774] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4804.757900] Lustre: lustre-OST0000: new disk, initializing [ 4804.767523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4804.834165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4808.251679] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4812.332295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4812.341812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4816.933379] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4820.385937] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4820.578234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4823.719217] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4823.919801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4828.132300] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4828.135056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4828.136556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4828.150312] Lustre: Skipped 1 previous similar message [ 4831.575131] Lustre: server umount lustre-OST0000 complete [ 4834.273215] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4834.282815] LustreError: Skipped 1 previous similar message [ 4834.286245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4834.297485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4834.302612] Lustre: Skipped 1 previous similar message [ 4838.369638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4838.372824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4840.304607] Lustre: server umount lustre-MDT0000 complete [ 4843.200914] LustreError: 269760:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761126059 with bad export cookie 10820777201612140480 [ 4843.202176] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4843.213213] LustreError: 269760:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4843.392385] Lustre: server umount lustre-MDT0001 complete [ 4849.806155] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 05:41:05 (1761126065) [ 4854.871760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4855.242782] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.305991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4858.341474] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4860.386849] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.201583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.581872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4869.654911] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4877.019609] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4882.567709] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4888.155447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4888.386521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4891.752054] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4897.670566] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4920.289162] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4920.292769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.295728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4920.298349] Lustre: Skipped 1 previous similar message [ 4920.308217] Lustre: Skipped 2 previous similar messages [ 4925.411895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4925.417870] Lustre: Skipped 1 previous similar message [ 4925.865497] Lustre: server umount lustre-OST0000 complete [ 4929.365972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4939.806171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4940.038072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4943.423943] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 4947.181713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4947.194087] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4947.199036] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 4947.212343] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 4947.221026] Lustre: Skipped 1 previous similar message [ 4947.232350] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 4949.580890] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4957.666805] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4957.669557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4957.679107] LustreError: Skipped 1 previous similar message [ 4957.681217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4957.697424] Lustre: Skipped 1 previous similar message [ 4957.706895] Lustre: Skipped 1 previous similar message [ 4963.444298] Lustre: server umount lustre-OST0000 complete [ 4967.904494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4972.749690] Lustre: server umount lustre-MDT0000 complete [ 4973.025770] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4973.032245] LustreError: Skipped 1 previous similar message [ 4975.629813] LustreError: 273440:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761126192 with bad export cookie 10820777201612141334 [ 4975.639133] LustreError: 273440:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4975.641119] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4984.913661] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 4987.251686] Key type lgssc unregistered [ 4987.441383] LNet: 278893:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4987.448254] LNet: Removed LNI 192.168.204.133@tcp [ 4988.359797] Key type .llcrypt unregistered [ 4988.361469] Key type ._llcrypt unregistered [ 5005.742535] alg: No test for adler32 (adler32-zlib) [ 5006.496743] Key type ._llcrypt registered [ 5006.499165] Key type .llcrypt registered [ 5006.584800] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5007.228867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5007.671259] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 5007.839571] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 5007.844807] LNet: Accept secure, port 988 [ 5009.495161] Key type lgssc registered [ 5010.275648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5016.955470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5018.272586] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5018.319122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5021.865200] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5023.718578] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5028.698062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5028.833129] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5029.041717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5032.468690] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5038.846635] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5044.255579] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5050.520703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5050.750632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5054.001453] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5060.218449] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5069.179431] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 5094.883869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.888500] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5094.896719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5094.902569] Lustre: Skipped 1 previous similar message [ 5094.930348] LustreError: Skipped 1 previous similar message [ 5097.428455] Lustre: server umount lustre-OST0000 complete [ 5105.120129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5105.125420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5105.128129] Lustre: Skipped 1 previous similar message [ 5110.242634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5110.245083] Lustre: Skipped 1 previous similar message [ 5115.359205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5115.475054] Lustre: server umount lustre-MDT0000 complete [ 5116.388307] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5116.413099] LustreError: Skipped 1 previous similar message [ 5118.736442] LustreError: 280350:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761126335 with bad export cookie 13926715150562383796 [ 5118.739595] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5118.743141] LustreError: 280350:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5119.030668] Lustre: server umount lustre-MDT0001 complete [ 5129.623859] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 5132.037393] Key type lgssc unregistered [ 5132.352994] LNet: 284819:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5132.360185] LNet: Removed LNI 192.168.204.133@tcp [ 5133.320637] Key type .llcrypt unregistered [ 5133.323283] Key type ._llcrypt unregistered [ 5149.192083] alg: No test for adler32 (adler32-zlib) [ 5149.946491] Key type ._llcrypt registered [ 5149.948457] Key type .llcrypt registered [ 5150.063662] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5151.064560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5151.632082] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 5151.887470] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 5151.890452] LNet: Accept secure, port 988 [ 5153.575142] Key type lgssc registered [ 5154.722631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5162.211610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5163.671403] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5163.729886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5167.920459] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5169.127061] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5174.249775] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.825954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5176.183049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5179.989925] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5186.608525] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5192.030447] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5197.939158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5198.352179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5201.821402] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5208.674607] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5213.741587] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 5223.904301] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5223.910524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.915168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5223.924016] Lustre: Skipped 1 previous similar message [ 5223.936317] Lustre: Skipped 1 previous similar message [ 5228.748680] Lustre: server umount lustre-OST0000 complete [ 5232.613416] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5232.628195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5232.643021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5234.144603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5234.147205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5238.291412] Lustre: server umount lustre-MDT0000 complete [ 5239.264995] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5239.274236] LustreError: Skipped 1 previous similar message [ 5241.548973] LustreError: 286088:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761126458 with bad export cookie 8365716085278970533 [ 5241.550520] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5241.565232] LustreError: 286088:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5241.788725] Lustre: server umount lustre-MDT0001 complete [ 5252.239188] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 5254.844405] Key type lgssc unregistered [ 5255.081896] LNet: 289931:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5255.089950] LNet: Removed LNI 192.168.204.133@tcp [ 5255.907113] Key type .llcrypt unregistered [ 5255.908762] Key type ._llcrypt unregistered [ 5270.412887] alg: No test for adler32 (adler32-zlib) [ 5271.168409] Key type ._llcrypt registered [ 5271.174631] Key type .llcrypt registered [ 5271.290560] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5272.299435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5272.749413] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 5272.946728] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 5272.952826] LNet: Accept secure, port 988 [ 5274.623402] Key type lgssc registered [ 5275.586844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5282.743077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.253909] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5284.323230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5288.499254] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5289.442341] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5294.563501] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5296.259484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5296.589519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5300.924160] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5309.042736] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5315.386965] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5322.173849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5322.621725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5326.950961] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5333.905815] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 5334.114653] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5342.178488] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5342.198902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5342.229495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5344.225141] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5344.250305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5344.257201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.269857] Lustre: Skipped 1 previous similar message [ 5347.887552] Lustre: server umount lustre-OST0000 complete [ 5352.931241] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5352.955495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.984634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5358.956505] Lustre: server umount lustre-MDT0000 complete [ 5359.584106] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5359.604669] LustreError: Skipped 1 previous similar message [ 5362.381023] LustreError: 291985:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761126578 with bad export cookie 5040044653242672070 [ 5362.383575] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5362.405456] LustreError: 291985:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5362.735562] Lustre: server umount lustre-MDT0001 complete [ 5374.557358] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 5377.068267] Key type lgssc unregistered [ 5377.360124] LNet: 294853:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5377.373339] LNet: Removed LNI 192.168.204.133@tcp [ 5378.311807] Key type .llcrypt unregistered [ 5378.313154] Key type ._llcrypt unregistered [ 5398.395798] alg: No test for adler32 (adler32-zlib) [ 5399.150252] Key type ._llcrypt registered [ 5399.158261] Key type .llcrypt registered [ 5399.284601] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5399.930765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5400.450965] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 5400.695496] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 5400.702289] LNet: Accept secure, port 988 [ 5402.343367] Key type lgssc registered [ 5403.383990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5410.808515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.263226] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5412.312595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.864190] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5417.447862] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.561247] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.764185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5423.228848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5426.583655] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5433.312060] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5439.484728] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5446.357631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5446.671110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5450.499091] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5457.633718] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5462.067295] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 5484.518613] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5484.534691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5484.544373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5487.584988] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5487.589835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5487.601337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5490.341903] Lustre: server umount lustre-OST0000 complete [ 5494.757122] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5494.760435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5494.767155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5494.771550] Lustre: Skipped 1 previous similar message [ 5497.826342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5497.829091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5497.853875] Lustre: Skipped 1 previous similar message [ 5500.007267] Lustre: server umount lustre-MDT0000 complete [ 5502.952233] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5502.966693] LustreError: Skipped 1 previous similar message [ 5503.657270] LustreError: 296312:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761126720 with bad export cookie 15874558314036981410 [ 5503.663396] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5503.674957] LustreError: 296312:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5504.061518] Lustre: server umount lustre-MDT0001 complete [ 5516.494128] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 5519.541976] Key type lgssc unregistered [ 5519.819644] LNet: 300810:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5519.831929] LNet: Removed LNI 192.168.204.133@tcp [ 5520.594901] Key type .llcrypt unregistered [ 5520.598716] Key type ._llcrypt unregistered [ 5536.653096] alg: No test for adler32 (adler32-zlib) [ 5537.407380] Key type ._llcrypt registered [ 5537.411116] Key type .llcrypt registered [ 5537.516809] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5538.937840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5539.453381] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 5539.723732] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 5539.730258] LNet: Accept secure, port 988 [ 5541.415138] Key type lgssc registered [ 5542.543597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5549.980891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5551.422206] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5551.493586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5555.361767] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5556.711113] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5561.833124] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5563.840350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5564.312812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5568.656594] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5578.488308] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5585.291249] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5593.590066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5593.862937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5598.402350] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5602.037933] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 5606.405875] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5617.632660] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5617.637891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5617.657178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5617.682458] Lustre: Skipped 1 previous similar message [ 5622.754614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5622.759306] Lustre: Skipped 1 previous similar message [ 5623.656425] Lustre: server umount lustre-OST0000 complete [ 5627.873063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.876175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5627.897063] Lustre: Skipped 1 previous similar message [ 5627.915434] Lustre: Skipped 2 previous similar messages [ 5632.994604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5632.999904] Lustre: Skipped 1 previous similar message [ 5633.979644] Lustre: server umount lustre-MDT0000 complete [ 5637.761962] LustreError: 304091:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761126854 with bad export cookie 9451563263471075180 [ 5637.764806] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5637.777373] LustreError: 304091:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5638.121991] Lustre: server umount lustre-MDT0001 complete [ 5649.357719] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 5653.024123] Key type lgssc unregistered [ 5653.355459] LNet: 305938:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5653.376157] LNet: Removed LNI 192.168.204.133@tcp [ 5654.315673] Key type .llcrypt unregistered [ 5654.325692] Key type ._llcrypt unregistered [ 5672.521987] alg: No test for adler32 (adler32-zlib) [ 5673.279982] Key type ._llcrypt registered [ 5673.284067] Key type .llcrypt registered [ 5673.370766] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5674.598147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5675.148721] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 5675.533871] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 5675.539896] LNet: Accept secure, port 988 [ 5677.191176] Key type lgssc registered [ 5678.272276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5684.816830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5686.149590] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5686.198365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5690.082126] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5691.372942] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5696.486242] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5697.760662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.205806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5702.156210] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5709.879496] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5714.874453] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5720.597460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5720.952279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.954487] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5732.642807] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5736.321406] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 5744.611072] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5744.619577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5744.626535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5746.661967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5746.666847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5746.669983] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5749.902780] Lustre: server umount lustre-OST0000 complete [ 5754.341730] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5754.344529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5754.369582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5754.380127] Lustre: Skipped 1 previous similar message [ 5756.897509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5756.900842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5756.911839] Lustre: Skipped 1 previous similar message [ 5759.913082] Lustre: server umount lustre-MDT0000 complete [ 5762.020158] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5762.031536] LustreError: Skipped 1 previous similar message [ 5764.179504] LustreError: 307208:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761126980 with bad export cookie 15479127806157752773 [ 5764.190180] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5764.216813] LustreError: 307208:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5764.618416] Lustre: server umount lustre-MDT0001 complete [ 5776.193416] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 5778.242552] Key type lgssc unregistered [ 5778.486824] LNet: 310883:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5778.498790] LNet: Removed LNI 192.168.204.133@tcp [ 5779.028247] Key type .llcrypt unregistered [ 5779.030097] Key type ._llcrypt unregistered [ 5789.053717] alg: No test for adler32 (adler32-zlib) [ 5789.805512] Key type ._llcrypt registered [ 5789.807143] Key type .llcrypt registered [ 5789.913762] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5804.421745] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 5817.896800] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5819.460815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5820.222967] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 5820.447306] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 5820.453387] LNet: Accept secure, port 988 [ 5822.127172] Key type lgssc registered [ 5823.372724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5834.509661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5842.025478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5849.307343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5856.524024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5863.860759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5863.950568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.164341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5865.201655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5865.276318] Lustre: lustre-MDT0000: new disk, initializing [ 5865.370931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5865.381282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5869.324134] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5879.075969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5879.147532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.203921] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5879.247152] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5879.250460] Lustre: Skipped 1 previous similar message [ 5879.333629] Lustre: lustre-MDT0001: new disk, initializing [ 5879.472799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5879.517961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5879.533477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5883.909879] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5893.522542] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5899.258988] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5906.002692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5906.124698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5906.142982] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5906.374267] Lustre: lustre-OST0000: new disk, initializing [ 5906.377171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5906.474495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5910.723698] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5912.645709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5912.661608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5919.243306] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5923.530583] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5923.714672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5927.377431] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5927.574784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5933.540720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.544975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5933.552096] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5933.580664] Lustre: Skipped 1 previous similar message [ 5933.594111] Lustre: Skipped 1 previous similar message [ 5935.629175] Lustre: server umount lustre-OST0000 complete [ 5941.216550] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5941.219263] LustreError: Skipped 1 previous similar message [ 5941.230410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5941.236864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5943.777519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5943.777851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5945.447408] Lustre: server umount lustre-MDT0000 complete [ 5948.897702] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5948.908722] LustreError: Skipped 1 previous similar message [ 5949.165781] LustreError: 314628:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761127165 with bad export cookie 6043238392197140935 [ 5949.166890] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5949.184874] LustreError: 314628:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5949.516346] Lustre: server umount lustre-MDT0001 complete [ 5957.772138] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:59:33 (1761127173) [ 5963.864599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5964.606543] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5964.698106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.396943] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5969.895902] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5975.008763] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5978.698352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.110822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5983.352824] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 5991.945736] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5998.930661] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6008.305204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6008.618070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6013.681304] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6022.451394] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6035.437517] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6035.440274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6035.446049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6035.448307] Lustre: Skipped 1 previous similar message [ 6037.328726] Lustre: server umount lustre-OST0000 complete [ 6045.665419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.673895] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6045.676223] Lustre: Skipped 1 previous similar message [ 6045.678354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6045.680917] LustreError: Skipped 1 previous similar message [ 6045.690340] Lustre: Skipped 2 previous similar messages [ 6047.433514] Lustre: server umount lustre-MDT0000 complete [ 6050.573076] LustreError: 319165:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761127267 with bad export cookie 6043238392197141789 [ 6050.573862] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6050.583706] LustreError: 319165:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6061.059623] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6063.799279] Key type lgssc unregistered [ 6064.112568] LNet: 322858:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6064.127219] LNet: Removed LNI 192.168.204.133@tcp [ 6064.888902] Key type .llcrypt unregistered [ 6064.890906] Key type ._llcrypt unregistered [ 6086.802614] alg: No test for adler32 (adler32-zlib) [ 6087.556617] Key type ._llcrypt registered [ 6087.558627] Key type .llcrypt registered [ 6087.669450] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 6095.563024] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6096.276236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6096.579827] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 6096.787457] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6096.791905] LNet: Accept secure, port 988 [ 6098.463342] Key type lgssc registered [ 6099.426277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6107.095036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6111.534666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6115.785352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6120.326919] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6124.088304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6127.557712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6132.230899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6132.273226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.404854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6133.425907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6133.479319] Lustre: lustre-MDT0000: new disk, initializing [ 6133.543297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6133.559307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6136.092540] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6144.084047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6144.125923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.191253] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6144.230333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6144.235177] Lustre: Skipped 1 previous similar message [ 6144.288905] Lustre: lustre-MDT0001: new disk, initializing [ 6144.354868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6144.389528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6144.399599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6147.596594] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6154.765814] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6159.102282] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6165.116932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6165.175194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6165.178450] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6165.388642] Lustre: lustre-OST2710: new disk, initializing [ 6165.404252] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 6165.530676] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6169.085437] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6171.714945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6171.722491] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6177.368160] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6183.109248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6183.187835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6183.192835] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6183.282771] Lustre: lustre-OST03e8: new disk, initializing [ 6183.287515] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 6183.356390] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6186.439571] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6192.826602] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6193.758974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6193.771470] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6197.717088] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 6197.873782] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6201.069459] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 6201.214475] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6204.081872] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 6204.207258] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6206.711427] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 6206.866216] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6209.507473] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6209.509638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6209.516111] Lustre: Skipped 3 previous similar messages [ 6209.518695] Lustre: Skipped 2 previous similar messages [ 6214.626527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6214.630300] Lustre: Skipped 4 previous similar messages [ 6214.788189] Lustre: server umount lustre-MDT0000 complete [ 6217.090772] LustreError: 326796:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761127433 with bad export cookie 18325365198801057739 [ 6217.095075] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6217.097522] LustreError: 326796:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6217.316575] Lustre: server umount lustre-MDT0001 complete [ 6225.887260] Lustre: 332289:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761127436/real 1761127436] req@00000000db5f9956 x1846675833384256/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761127442 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6225.921825] Lustre: server umount lustre-OST2710 complete [ 6226.912192] Lustre: 324598:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761127436/real 1761127436] req@000000009b2cbbbf x1846675833384384/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761127443 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6226.925224] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.482992] Lustre: server umount lustre-OST03e8 complete [ 6242.194459] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 6248.398282] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6255.565509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6259.483787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6263.155449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6267.205430] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6271.455467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6271.501608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6271.673935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6271.706754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6271.756736] Lustre: lustre-MDT0000: new disk, initializing [ 6271.831662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6271.846917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6274.373513] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6280.967196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6281.012408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.070474] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6281.102175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6281.105870] Lustre: Skipped 1 previous similar message [ 6281.168680] Lustre: lustre-MDT0001: new disk, initializing [ 6281.227646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6281.251169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6281.259904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6284.020056] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6291.141110] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6294.879209] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6299.124112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6299.188054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6299.192786] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6299.341320] Lustre: lustre-OST0000: new disk, initializing [ 6299.344044] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6299.400922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6302.060761] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6307.351821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6307.356376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6308.151960] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6311.318537] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6311.502163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6314.087608] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6314.258157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6317.541389] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6317.541877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6317.545602] LustreError: Skipped 1 previous similar message [ 6317.556345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6317.561707] Lustre: Skipped 1 previous similar message [ 6321.735519] Lustre: server umount lustre-OST0000 complete [ 6327.264898] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6327.270217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6327.277353] Lustre: Skipped 1 previous similar message [ 6327.283406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6330.530287] Lustre: server umount lustre-MDT0000 complete [ 6332.896937] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.907307] LustreError: Skipped 1 previous similar message [ 6333.342175] LustreError: 337480:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761127549 with bad export cookie 18325365198801059349 [ 6333.347452] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.351793] LustreError: 337480:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6339.517371] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 06:05:55 (1761127555) [ 6340.478992] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6341.570760] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 06:05:57 (1761127557) [ 6344.166109] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6350.687840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6353.364423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6355.907179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6358.432451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6359.486637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6364.717366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6364.881908] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6364.898727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6365.016044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6367.567397] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6372.171413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.219601] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6374.886731] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6379.449036] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6382.643414] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6386.468406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6386.517315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6386.632241] Lustre: Denying initial registration attempt from nid 192.168.204.133@tcp, specified as failover [ 6386.637866] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6386.642462] LustreError: 343413:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 6386.647593] LustreError: 343413:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 6386.650861] LustreError: 343413:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 6386.728556] Lustre: server umount lustre-OST0000 complete [ 6386.730731] Lustre: Skipped 1 previous similar message [ 6386.732614] LustreError: 343413:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6389.733847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6389.736354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6389.741937] Lustre: Skipped 2 previous similar messages [ 6389.748184] Lustre: Skipped 1 previous similar message [ 6397.941467] LustreError: 342258:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761127614 with bad export cookie 18325365198801060210 [ 6397.944274] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6397.945940] LustreError: 342258:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6405.219118] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6406.668787] Key type lgssc unregistered [ 6406.837236] LNet: 344485:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6406.844804] LNet: Removed LNI 192.168.204.133@tcp [ 6407.323845] Key type .llcrypt unregistered [ 6407.328730] Key type ._llcrypt unregistered [ 6413.911534] alg: No test for adler32 (adler32-zlib) [ 6414.663471] Key type ._llcrypt registered [ 6414.665193] Key type .llcrypt registered [ 6414.719362] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6415.276210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6415.461870] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 6415.564819] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6415.567716] LNet: Accept secure, port 988 [ 6417.184256] Key type lgssc registered [ 6417.795320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6421.646493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6423.972449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6425.987125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6428.236778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6428.913798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6437.833613] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6443.396815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6444.606646] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6444.618241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6444.735385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6447.334479] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6452.564142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6452.620500] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6452.647884] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6452.788394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6455.480176] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6460.384707] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6463.826276] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6467.806890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6467.843814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6467.949549] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6468.053698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6471.095984] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6475.520318] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6480.866959] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6480.867634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6480.870244] LustreError: Skipped 1 previous similar message [ 6480.876072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6480.879712] Lustre: Skipped 1 previous similar message [ 6482.615461] Lustre: server umount lustre-OST0000 complete [ 6485.985197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6485.999090] Lustre: Skipped 1 previous similar message [ 6486.005102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6486.007737] Lustre: Skipped 1 previous similar message [ 6491.105551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6491.111462] Lustre: Skipped 1 previous similar message [ 6496.227575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6496.231399] Lustre: Skipped 1 previous similar message [ 6499.295270] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6499.361430] Lustre: server umount lustre-MDT0000 complete [ 6501.624324] LustreError: 347353:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761127718 with bad export cookie 18273875528637814832 [ 6501.627530] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6501.630416] LustreError: 347353:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6501.801954] Lustre: server umount lustre-MDT0001 complete [ 6506.781810] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6508.113953] Key type lgssc unregistered [ 6508.260390] LNet: 351055:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6508.264332] LNet: Removed LNI 192.168.204.133@tcp [ 6508.631982] Key type .llcrypt unregistered [ 6508.633917] Key type ._llcrypt unregistered [ 6520.170626] alg: No test for adler32 (adler32-zlib) [ 6520.922575] Key type ._llcrypt registered [ 6520.924328] Key type .llcrypt registered [ 6520.974119] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6521.378924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6521.648360] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 6521.744879] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6521.747690] LNet: Accept secure, port 988 [ 6523.343144] Key type lgssc registered [ 6523.800876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6527.603891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6528.818596] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6528.857413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6530.834704] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6534.114123] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.189073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.384310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6537.479980] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6541.606072] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6544.547816] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6547.935279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6548.132189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6550.194712] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6554.186918] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6564.699809] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 6564.833262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6564.840485] Lustre: Skipped 2 previous similar messages [ 6568.851254] Lustre: server umount lustre-MDT0000 complete [ 6569.811526] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6570.823325] LustreError: 352496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761127787 with bad export cookie 1362163302669919406 [ 6570.827175] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6570.831179] LustreError: 352496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6574.932108] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6574.947508] LustreError: Skipped 3 previous similar messages [ 6574.947950] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 6574.953707] Lustre: Skipped 3 previous similar messages [ 6575.074436] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6576.941858] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 6576.950382] Lustre: Skipped 1 previous similar message [ 6582.101581] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6582.102118] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 6582.110523] LustreError: Skipped 3 previous similar messages [ 6582.111939] Lustre: Skipped 1 previous similar message [ 6585.311703] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6585.423923] Lustre: server umount lustre-MDT0001 complete [ 6588.639242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6592.493316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6603.681442] Lustre: Evicted from MGS (at 192.168.204.133@tcp) after server handle changed from 0x12e7604e85b6a4ae to 0x12e7604e85b6c0a0 [ 6603.687270] Lustre: MGC192.168.204.133@tcp: Connection restored to 192.168.204.133@tcp (at 0@lo) [ 6603.827954] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6603.835352] LustreError: Skipped 2 previous similar messages [ 6603.870072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6603.913969] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 6606.208440] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6611.162801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.340493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6612.389353] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 6613.327865] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6617.057107] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6617.831622] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6620.507803] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6621.355927] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6624.224703] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6624.228878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6624.235718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6624.239969] Lustre: Skipped 2 previous similar messages [ 6626.785316] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6626.789112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6629.431967] Lustre: server umount lustre-OST0000 complete [ 6631.904414] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6631.907974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6632.928206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.931037] Lustre: Skipped 5 previous similar messages [ 6637.360970] Lustre: server umount lustre-MDT0000 complete [ 6638.048898] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6638.054375] LustreError: Skipped 3 previous similar messages [ 6639.200038] LustreError: 352496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761127855 with bad export cookie 1362163302669926560 [ 6639.202718] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6639.205196] LustreError: 352496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6645.480773] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6646.843138] Key type lgssc unregistered [ 6646.986854] LNet: 359008:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6646.993699] LNet: Removed LNI 192.168.204.133@tcp [ 6647.363187] Key type .llcrypt unregistered [ 6647.364812] Key type ._llcrypt unregistered [ 6658.378154] alg: No test for adler32 (adler32-zlib) [ 6659.130500] Key type ._llcrypt registered [ 6659.131988] Key type .llcrypt registered [ 6659.178475] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6659.688198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6659.856842] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 6659.955113] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6659.957463] LNet: Accept secure, port 988 [ 6661.551338] Key type lgssc registered [ 6662.003285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6665.522939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6666.747424] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6666.784569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6668.790677] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6671.840585] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6673.089815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.264688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6675.287291] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6678.500691] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6680.946165] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6683.618173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6683.618979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6683.623914] Lustre: Skipped 1 previous similar message [ 6683.631075] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6687.712801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6687.716652] Lustre: Skipped 2 previous similar messages [ 6688.407429] Lustre: server umount lustre-MDT0000 complete [ 6690.168204] LustreError: 360448:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761127906 with bad export cookie 1092693604468802742 [ 6690.172987] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6690.174016] LustreError: 360448:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6690.319395] Lustre: server umount lustre-MDT0001 complete [ 6702.694251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6703.341586] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6703.402943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6703.617270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6709.704771] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6719.565849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6719.650123] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6719.698441] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6719.968993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6724.758659] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6734.817509] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6741.995796] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6752.038916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6752.314182] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6752.335493] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6752.357977] LustreError: 365160:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 6752.384488] LustreError: 365160:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 6752.387588] LustreError: 365160:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 6752.583322] Lustre: server umount lustre-OST0000 complete [ 6752.585560] LustreError: 365160:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6760.551493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6760.883148] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6761.175867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6765.173461] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6768.738655] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 6771.952093] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6778.406743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6778.484186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6778.487762] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6778.535623] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6778.598369] Lustre: lustre-OST0001: new disk, initializing [ 6778.602700] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6778.690287] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6782.465486] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6787.170147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6787.183590] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6788.904679] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6792.678030] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6792.681119] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6792.689490] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6792.710744] Lustre: Skipped 1 previous similar message [ 6793.463253] Lustre: server umount lustre-OST0001 complete [ 6797.796374] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.801640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6797.812180] LustreError: Skipped 1 previous similar message [ 6797.822234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6797.822243] Lustre: Skipped 1 previous similar message [ 6802.913810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6802.917903] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6802.927464] Lustre: Skipped 2 previous similar messages [ 6808.032907] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6808.048577] LustreError: Skipped 2 previous similar messages [ 6811.105054] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6811.242816] Lustre: server umount lustre-OST0000 complete [ 6817.248161] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6817.251632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.260450] Lustre: Skipped 1 previous similar message [ 6817.263789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6817.267622] Lustre: Skipped 2 previous similar messages [ 6820.198274] Lustre: server umount lustre-MDT0000 complete [ 6823.394883] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.409090] LustreError: Skipped 1 previous similar message [ 6823.484178] LustreError: 363156:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761128040 with bad export cookie 1092693604468803281 [ 6823.485741] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6823.495429] LustreError: 363156:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6833.261733] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6835.685795] Key type lgssc unregistered [ 6835.915772] LNet: 368528:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6835.922709] LNet: Removed LNI 192.168.204.133@tcp [ 6836.620930] Key type .llcrypt unregistered [ 6836.623544] Key type ._llcrypt unregistered [ 6864.313284] alg: No test for adler32 (adler32-zlib) [ 6865.071107] Key type ._llcrypt registered [ 6865.073595] Key type .llcrypt registered [ 6865.180886] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 6877.981592] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6878.824319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6879.158642] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 6879.311352] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6879.313831] LNet: Accept secure, port 988 [ 6880.927195] Key type lgssc registered [ 6881.882103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6889.480793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6895.028398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6899.777730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6905.215774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6910.524367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6910.569330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6911.773051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6911.805747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6911.888952] Lustre: lustre-MDT0000: new disk, initializing [ 6911.972634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6911.997102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6915.308333] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6924.189826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6924.256858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6924.318805] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6924.375106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6924.380058] Lustre: Skipped 1 previous similar message [ 6924.438669] Lustre: lustre-MDT0001: new disk, initializing [ 6924.520320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6924.560679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6924.572196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6928.365338] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6937.854152] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6942.784825] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6947.904412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6947.959172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6947.963908] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6948.156825] Lustre: lustre-OST0000: new disk, initializing [ 6948.160496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6948.230731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6951.474463] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 6960.597183] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6964.699494] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6964.917719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6968.511095] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6968.650920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6971.364482] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6971.365474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6971.366521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6971.385519] Lustre: Skipped 1 previous similar message [ 6976.247391] Lustre: server umount lustre-OST0000 complete [ 6981.088106] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6981.091600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.119894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6981.122613] Lustre: Skipped 1 previous similar message [ 6985.760943] Lustre: server umount lustre-MDT0000 complete [ 6986.720658] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6986.747547] LustreError: Skipped 1 previous similar message [ 6989.111615] LustreError: 374725:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761128205 with bad export cookie 13543092656697957182 [ 6989.118363] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6989.140053] LustreError: 374725:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6989.375439] Lustre: server umount lustre-MDT0001 complete [ 6996.964556] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 06:16:52 (1761128212) [ 7001.955647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7007.006775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7026.443852] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 7033.949565] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 7042.541447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7048.200136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7053.583221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7059.630259] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7064.724637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7064.795426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7064.958610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7064.987602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7065.038094] Lustre: lustre-MDT0000: new disk, initializing [ 7065.109938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7065.119418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7068.695841] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7077.487665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7077.527716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7077.589421] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7077.621550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7077.624314] Lustre: Skipped 1 previous similar message [ 7077.711823] Lustre: lustre-MDT0001: new disk, initializing [ 7077.787739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7077.827176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7077.833123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7081.072573] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7089.290294] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7093.839376] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7098.632777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7098.686918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7098.691764] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7098.892711] Lustre: lustre-OST0000: new disk, initializing [ 7098.899060] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7098.976830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7101.788176] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7108.720562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7108.732457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7109.269204] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7112.681359] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7112.852771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7116.614526] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7116.768201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7119.328076] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7119.334024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7119.335094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7119.348620] Lustre: Skipped 2 previous similar messages [ 7119.358644] Lustre: Skipped 1 previous similar message [ 7124.312620] Lustre: server umount lustre-OST0000 complete [ 7129.056492] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7129.060488] LustreError: Skipped 1 previous similar message [ 7129.073265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7129.095516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7129.098843] Lustre: Skipped 1 previous similar message [ 7134.691064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7134.694107] Lustre: Skipped 2 previous similar messages [ 7141.343127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7141.466036] Lustre: server umount lustre-MDT0000 complete [ 7144.931690] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.180176] LustreError: 380399:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761128361 with bad export cookie 13543092656697958036 [ 7145.192515] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7145.197196] LustreError: 380399:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7153.753644] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 06:19:29 (1761128369) [ 7161.049559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7161.435030] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7161.457869] LustreError: Skipped 1 previous similar message [ 7161.537176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7165.797453] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7166.945241] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7172.070269] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7173.702255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7178.603230] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7185.892918] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7191.689777] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7197.431389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7197.775925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7197.778593] Lustre: Skipped 1 previous similar message [ 7201.246583] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7208.470786] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7220.828628] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 7222.828572] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 7224.461394] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 7226.143992] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 7228.228269] Lustre: Failing over lustre-MDT0000 [ 7228.415633] Lustre: server umount lustre-MDT0000 complete [ 7228.456592] Lustre: Skipped 1 previous similar message [ 7229.810937] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7230.435250] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7230.441629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7230.464519] Lustre: Skipped 1 previous similar message [ 7240.161977] Lustre: 370861:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761128449/real 1761128449] req@000000002e139921 x1846676653393664/t0(0) o400->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761128456 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7240.186361] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7246.306292] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.328621] LustreError: Skipped 19 previous similar messages [ 7247.555484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.555774] Lustre: Evicted from MGS (at 192.168.204.133@tcp) after server handle changed from 0xbbf2bb0452aee1ea to 0xbbf2bb0452aee5da [ 7256.579618] Lustre: MGC192.168.204.133@tcp: Connection restored to 192.168.204.133@tcp (at 0@lo) [ 7256.919794] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7258.470191] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7261.524492] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7262.206299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 7262.245451] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7262.302541] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 7270.977358] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7272.674124] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7276.026604] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 7277.866170] Lustre: Failing over lustre-MDT0000 [ 7278.030341] Lustre: server umount lustre-MDT0000 complete [ 7278.987226] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7279.006974] LustreError: Skipped 8 previous similar messages [ 7282.661426] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7282.671294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.695531] Lustre: Skipped 3 previous similar messages [ 7289.826365] Lustre: 370864:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761128499/real 1761128499] req@00000000a17c33e5 x1846676653403968/t0(0) o400->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761128506 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7289.860124] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7297.016704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7307.256553] Lustre: Evicted from MGS (at 192.168.204.133@tcp) after server handle changed from 0xbbf2bb0452aee5da to 0xbbf2bb0452aee8e3 [ 7307.284157] Lustre: MGC192.168.204.133@tcp: Connection restored to (at 0@lo) [ 7307.286605] Lustre: Skipped 2 previous similar messages [ 7307.634743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7307.637550] Lustre: Skipped 1 previous similar message [ 7307.695034] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7308.760473] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7311.744930] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7312.892689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 7312.934859] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7312.999213] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 7321.581829] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7323.083422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7328.226224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7328.238299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7328.248204] Lustre: Skipped 2 previous similar messages [ 7328.252047] Lustre: Skipped 2 previous similar messages [ 7334.458364] Lustre: server umount lustre-MDT0000 complete [ 7338.031423] LustreError: 385036:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761128554 with bad export cookie 13543092656697960675 [ 7338.037589] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7338.059544] LustreError: 385036:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7347.681689] Lustre: 390868:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761128558/real 1761128558] req@0000000024b19c18 x1846676653415744/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761128564 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7360.011935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7360.354739] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7360.366893] LustreError: Skipped 30 previous similar messages [ 7363.990379] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7370.134271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.612318] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7379.997344] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7384.321733] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7388.972834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7391.276023] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 7392.558799] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7398.921602] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7410.658236] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7410.673312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7410.674882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7410.697617] Lustre: Skipped 2 previous similar messages [ 7410.735251] Lustre: Skipped 6 previous similar messages [ 7414.324365] Lustre: server umount lustre-OST0000 complete [ 7414.328200] Lustre: Skipped 2 previous similar messages [ 7426.370636] LustreError: 394548:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761128642 with bad export cookie 13543092656697961403 [ 7426.374257] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7426.379826] LustreError: 394548:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7437.188660] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 7439.745262] Key type lgssc unregistered [ 7440.003195] LNet: 395352:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7440.019121] LNet: Removed LNI 192.168.204.133@tcp [ 7440.705048] Key type .llcrypt unregistered [ 7440.708236] Key type ._llcrypt unregistered [ 7462.366773] alg: No test for adler32 (adler32-zlib) [ 7463.122453] Key type ._llcrypt registered [ 7463.125615] Key type .llcrypt registered [ 7463.204407] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 7464.013808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7464.382878] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 7464.587534] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 7464.591825] LNet: Accept secure, port 988 [ 7466.247200] Key type lgssc registered [ 7467.204985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7473.709514] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7473.713954] LustreError: 396838:0:(osd_handler.c:8211:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7473.719540] LustreError: 396838:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7473.723316] LustreError: 396838:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7473.728062] LustreError: 396838:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7473.740463] LustreError: 396838:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7479.586415] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7479.590216] LustreError: 397235:0:(osd_handler.c:8211:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7479.596021] LustreError: 397235:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7479.601933] LustreError: 397235:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7479.606173] LustreError: 397235:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7479.610770] LustreError: 397235:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7493.644534] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 7495.885361] Key type lgssc unregistered [ 7496.195063] LNet: 398192:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7496.203283] LNet: Removed LNI 192.168.204.133@tcp [ 7496.923827] Key type .llcrypt unregistered [ 7496.925324] Key type ._llcrypt unregistered [ 7512.864838] alg: No test for adler32 (adler32-zlib) [ 7513.617458] Key type ._llcrypt registered [ 7513.619064] Key type .llcrypt registered [ 7513.720710] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 7525.647627] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 7526.329893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7526.689303] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 7526.865909] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 7526.869816] LNet: Accept secure, port 988 [ 7528.519144] Key type lgssc registered [ 7529.260659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7536.868232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7542.984515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7548.263319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7553.385765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7558.908422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7558.980835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.174741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7560.192853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7560.274918] Lustre: lustre-MDT0000: new disk, initializing [ 7560.351303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7560.366661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7563.755044] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7572.024626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7572.085904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7572.148165] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7572.185524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7572.190643] Lustre: Skipped 1 previous similar message [ 7572.250442] Lustre: lustre-MDT0001: new disk, initializing [ 7572.325492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7572.352809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7572.373190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7575.665934] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7583.249981] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7588.260500] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7593.853970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7593.909356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7593.915516] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7594.095820] Lustre: lustre-OST0000: new disk, initializing [ 7594.100934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7594.164691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7597.285116] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7601.721338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7601.730991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7605.296096] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7609.010873] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7609.199422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7613.004083] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7613.216966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7617.512873] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7617.515492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7617.519524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7617.533871] Lustre: Skipped 1 previous similar message [ 7620.933414] Lustre: server umount lustre-OST0000 complete [ 7627.747533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7627.754447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7627.757660] Lustre: Skipped 1 previous similar message [ 7627.769138] Lustre: Skipped 1 previous similar message [ 7630.456403] Lustre: server umount lustre-MDT0000 complete [ 7632.864854] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7632.879111] LustreError: Skipped 1 previous similar message [ 7633.862833] LustreError: 401463:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761128850 with bad export cookie 9516053274478654550 [ 7633.863348] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7633.872327] LustreError: 401463:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7634.155539] Lustre: server umount lustre-MDT0001 complete [ 7641.330192] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7642.732883] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:27:38 (1761128858) [ 7648.139776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7648.515503] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7648.570474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7652.173541] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7653.856156] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7658.977637] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7659.483331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.742100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7663.556676] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7670.437732] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7675.248187] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7680.894727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7681.159670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7684.475445] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7690.881989] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7695.554696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7695.602121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7695.609538] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7695.683197] Lustre: lustre-OST0001: new disk, initializing [ 7695.688186] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7698.809219] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7703.110046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7703.121900] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7707.091122] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7713.636695] Lustre: lustre-OST0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 7713.642102] Lustre: Skipped 1 previous similar message [ 7713.761904] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7713.775242] Lustre: Skipped 1 previous similar message [ 7716.358888] Lustre: server umount lustre-OST0001 complete [ 7718.739070] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7727.016052] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7727.022236] LustreError: Skipped 5 previous similar messages [ 7731.681758] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7731.687273] LustreError: Skipped 1 previous similar message [ 7731.690829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7731.700072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7731.702702] Lustre: Skipped 2 previous similar messages [ 7737.173527] Lustre: server umount lustre-OST0000 complete [ 7741.921128] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7741.927649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.939728] Lustre: Skipped 1 previous similar message [ 7741.944648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7741.947975] Lustre: Skipped 2 previous similar messages [ 7746.246905] Lustre: server umount lustre-MDT0000 complete [ 7747.553142] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7747.565018] LustreError: Skipped 3 previous similar messages [ 7749.246714] LustreError: 406049:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761128965 with bad export cookie 9516053274478655404 [ 7749.253487] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7749.258223] LustreError: 406049:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7764.446858] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 7766.909755] Key type lgssc unregistered [ 7767.147119] LNet: 410972:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7767.160358] LNet: Removed LNI 192.168.204.133@tcp [ 7767.927631] Key type .llcrypt unregistered [ 7767.928809] Key type ._llcrypt unregistered [ 7792.965766] alg: No test for adler32 (adler32-zlib) [ 7793.725085] Key type ._llcrypt registered [ 7793.726905] Key type .llcrypt registered [ 7793.844815] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 7803.614738] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 7804.240971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7804.480246] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 7804.649761] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 7804.652669] LNet: Accept secure, port 988 [ 7806.279148] Key type lgssc registered [ 7807.057985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7812.527524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7817.318555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7821.444418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7825.126440] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7829.044556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7829.080090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7830.245937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7830.269606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7830.318336] Lustre: lustre-MDT0000: new disk, initializing [ 7830.364283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7830.374870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7832.895756] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7840.333115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7840.396230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7840.514877] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7840.597205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7840.602398] Lustre: Skipped 1 previous similar message [ 7840.652162] Lustre: lustre-MDT0001: new disk, initializing [ 7840.756592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7840.817450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7840.838428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7844.545609] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7851.962613] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7856.021837] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7860.872131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7860.927610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7860.931817] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7861.077666] Lustre: lustre-OST0000: new disk, initializing [ 7861.081651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7861.143534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7864.523896] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7867.440459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7867.447078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7872.317955] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7875.328064] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7875.468083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7878.421956] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7878.579820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7883.235373] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7883.235398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.240149] LustreError: Skipped 1 previous similar message [ 7883.246429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7883.250554] Lustre: Skipped 1 previous similar message [ 7886.151355] Lustre: server umount lustre-OST0000 complete [ 7891.941370] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7891.950623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7891.964299] Lustre: Skipped 1 previous similar message [ 7891.979727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7893.472028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7893.476054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7894.850959] Lustre: server umount lustre-MDT0000 complete [ 7897.368866] LustreError: 415913:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761129113 with bad export cookie 10918343524252153141 [ 7897.371215] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7897.375052] LustreError: 415913:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7897.580414] Lustre: server umount lustre-MDT0001 complete [ 7902.864633] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 06:31:58 (1761129118) [ 7908.497632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7914.126498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7914.379977] Lustre: lustre-MDT0000: new disk, initializing [ 7914.397066] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7914.447364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7917.030895] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7919.584385] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7921.776078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.965861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7924.652917] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7930.367297] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7934.574019] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7936.031141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7936.037366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7936.040065] Lustre: Skipped 1 previous similar message [ 7942.112623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7942.120481] Lustre: Skipped 4 previous similar messages [ 7950.304763] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7950.374315] Lustre: server umount lustre-MDT0000 complete [ 7952.360475] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7952.371558] LustreError: Skipped 1 previous similar message [ 7953.049592] LustreError: 420188:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761129169 with bad export cookie 10918343524252153995 [ 7953.052796] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7953.070121] LustreError: 420188:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7960.106913] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:32:55 (1761129175) [ 7964.226883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7964.464246] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7964.518373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7967.080975] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7969.762911] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7972.222286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7975.200272] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7980.049145] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7983.544208] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7987.913270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7988.110872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7988.113441] Lustre: Skipped 1 previous similar message [ 7990.772960] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 7995.408085] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8000.758984] Lustre: Permanently deactivating lustre-OST0000 [ 8000.771426] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 8006.651712] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8008.104059] LustreError: 425851:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 8008.109170] LustreError: 425851:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8008.112671] LustreError: 425851:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8009.188596] LustreError: 425900:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8009.191913] LustreError: 425900:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8016.244346] Lustre: server umount lustre-OST0000 complete [ 8016.246447] Lustre: Skipped 1 previous similar message [ 8017.744911] LustreError: 426149:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8017.750216] LustreError: 426149:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8020.960804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8020.961703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8020.966141] Lustre: Skipped 2 previous similar messages [ 8020.969021] Lustre: Skipped 2 previous similar messages [ 8025.058647] Lustre: 426246:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761129235/real 1761129235] req@00000000ccc490a4 x1846677624366464/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761129241 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8025.173337] Lustre: server umount lustre-MDT0000 complete [ 8026.080732] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8026.088093] LustreError: Skipped 1 previous similar message [ 8027.519886] LustreError: 422784:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761129244 with bad export cookie 10918343524252154548 [ 8027.521656] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8027.525146] LustreError: 422784:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8033.759188] Lustre: 426448:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761129244/real 1761129244] req@0000000044986515 x1846677624367424/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761129250 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8039.205839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8041.592358] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8043.130673] LustreError: 427356:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 8046.494289] Lustre: 427702:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.133@tcp [ 8053.728045] Lustre: 427801:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761129264/real 1761129264] req@00000000f0d204e3 x1846677624367872/t0(0) o251->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761129270 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8053.754802] Lustre: server umount MGS complete [ 8053.756850] Lustre: Skipped 1 previous similar message [ 8060.029168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8060.271261] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8060.283157] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8060.286021] Lustre: Skipped 1 previous similar message [ 8060.319949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8062.643318] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8067.183503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.319721] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8069.509250] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8073.433554] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8076.508632] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8077.478526] Lustre: Permanently reactivating lustre-OST0000 [ 8077.491255] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 8077.494348] Lustre: Skipped 2 previous similar messages [ 8099.705170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8102.460697] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8106.754516] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8114.658647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.660785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.664310] Lustre: Skipped 1 previous similar message [ 8114.664663] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8114.672130] Lustre: Skipped 1 previous similar message [ 8116.606158] Lustre: server umount lustre-OST0000 complete [ 8118.752649] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8118.756771] LustreError: Skipped 1 previous similar message [ 8133.087177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8135.384364] LustreError: 428409:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761129351 with bad export cookie 10918343524252155507 [ 8135.385032] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8135.388711] LustreError: 428409:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8142.556948] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8144.158402] Key type lgssc unregistered [ 8144.321136] LNet: 432814:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8144.326653] LNet: Removed LNI 192.168.204.133@tcp [ 8144.769206] Key type .llcrypt unregistered [ 8144.771084] Key type ._llcrypt unregistered [ 8154.668610] alg: No test for adler32 (adler32-zlib) [ 8155.421639] Key type ._llcrypt registered [ 8155.422852] Key type .llcrypt registered [ 8155.475756] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 8163.894453] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8164.359809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8164.680901] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 8164.812148] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8164.817048] LNet: Accept secure, port 988 [ 8166.423412] Key type lgssc registered [ 8166.999743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8172.183333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8176.361797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8180.084555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8183.736762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8187.822609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8187.873322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.084500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8189.101388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8189.146701] Lustre: lustre-MDT0000: new disk, initializing [ 8189.201121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.214908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8191.698536] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8198.219285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8198.260795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8198.312615] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8198.342067] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8198.344258] Lustre: Skipped 1 previous similar message [ 8198.386932] Lustre: lustre-MDT0001: new disk, initializing [ 8198.468714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8198.495603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8198.504407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8201.046228] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8207.219104] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8211.672573] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8217.242723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8217.307957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8217.311762] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8217.495318] Lustre: lustre-OST0000: new disk, initializing [ 8217.498943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8217.550081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8220.530459] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8225.318881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8225.328228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8227.105221] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8230.018388] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8230.140402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8232.597683] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8232.699226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8235.488573] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8235.497802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8235.507415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8235.511894] Lustre: Skipped 1 previous similar message [ 8240.009666] Lustre: server umount lustre-OST0000 complete [ 8244.705041] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8244.711480] LustreError: Skipped 1 previous similar message [ 8244.716185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8244.721634] Lustre: Skipped 1 previous similar message [ 8244.726836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8245.733090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8245.734646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8248.677626] Lustre: server umount lustre-MDT0000 complete [ 8250.853320] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8250.861406] LustreError: Skipped 1 previous similar message [ 8251.365985] LustreError: 436081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761129467 with bad export cookie 6196127445914982602 [ 8251.369355] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8251.374095] LustreError: 436081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8251.573946] Lustre: server umount lustre-MDT0001 complete [ 8256.741774] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8257.606095] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:37:53 (1761129473) [ 8261.858280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8262.097818] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8262.154231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8264.496332] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8267.232730] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8269.379424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.570081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8271.653819] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8275.632858] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8278.689227] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8282.664076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8282.827139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8285.074966] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8289.391307] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8290.506836] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8296.955306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8300.516657] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8300.519100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.528411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8300.530962] Lustre: Skipped 1 previous similar message [ 8302.565714] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8305.470495] Lustre: server umount lustre-OST0000 complete [ 8307.680351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8307.680891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8307.686985] Lustre: Skipped 2 previous similar messages [ 8307.689552] Lustre: Skipped 2 previous similar messages [ 8313.684676] Lustre: server umount lustre-MDT0000 complete [ 8315.533597] LustreError: 440668:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761129532 with bad export cookie 6196127445914983442 [ 8315.536412] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8315.538659] LustreError: 440668:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8322.870629] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8324.425558] Key type lgssc unregistered [ 8324.612231] LNet: 444532:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8324.620346] LNet: Removed LNI 192.168.204.133@tcp [ 8325.090783] Key type .llcrypt unregistered [ 8325.092691] Key type ._llcrypt unregistered [ 8338.838217] alg: No test for adler32 (adler32-zlib) [ 8339.590394] Key type ._llcrypt registered [ 8339.592123] Key type .llcrypt registered [ 8339.660548] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8339.812165] Key type .llcrypt unregistered [ 8339.813587] Key type ._llcrypt unregistered [ 8349.156295] alg: No test for adler32 (adler32-zlib) [ 8349.908429] Key type ._llcrypt registered [ 8349.909837] Key type .llcrypt registered [ 8349.990685] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8350.497660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8350.781863] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 8350.896029] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8350.900363] LNet: Accept secure, port 988 [ 8352.519550] Key type lgssc registered [ 8353.121826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8357.601786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8358.856474] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8358.897041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8361.063053] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8364.000815] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8365.533365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8365.696417] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8365.711981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8367.667820] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8371.170422] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.176670] LustreError: Skipped 2 previous similar messages [ 8371.348730] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8374.246606] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 8374.363347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8374.520154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8376.391718] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8384.994188] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8384.997317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8385.011705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8389.603712] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8389.614946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8389.625695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8390.834956] Lustre: server umount lustre-OST0000 complete [ 8394.722832] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8394.726273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8394.732494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8394.735368] Lustre: Skipped 2 previous similar messages [ 8394.843197] Lustre: server umount lustre-MDT0000 complete [ 8395.233811] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8395.241113] LustreError: Skipped 3 previous similar messages [ 8396.768995] LustreError: 449115:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761129613 with bad export cookie 11836733120752435807 [ 8396.771642] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8396.773833] LustreError: 449115:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8396.955473] Lustre: server umount lustre-MDT0001 complete [ 8403.237686] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8404.672956] Key type lgssc unregistered [ 8404.858485] LNet: 450121:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8404.864992] LNet: Removed LNI 192.168.204.133@tcp [ 8405.259968] Key type .llcrypt unregistered [ 8405.261669] Key type ._llcrypt unregistered [ 8418.491298] alg: No test for adler32 (adler32-zlib) [ 8419.243415] Key type ._llcrypt registered [ 8419.245388] Key type .llcrypt registered [ 8419.311618] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8419.935632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8420.156601] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 8420.269937] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8420.272488] LNet: Accept secure, port 988 [ 8421.879151] Key type lgssc registered [ 8422.459087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8426.524842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8443.039669] LustreError: 451556:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.133@tcp: failed processing log, type 1: rc = -5 [ 8474.783780] LustreError: 451556:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.133@tcp: failed processing log, type 4: rc = -110 [ 8504.479207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8506.514679] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8510.673920] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8514.116759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.196548] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.248687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8527.287015] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 8529.375432] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8532.449840] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8533.902380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8534.105606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8536.232600] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8540.790491] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8544.561290] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8548.321390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8548.325474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8548.328526] Lustre: Skipped 1 previous similar message [ 8553.441259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8553.445506] Lustre: Skipped 1 previous similar message [ 8554.485686] Lustre: server umount lustre-OST0000 complete [ 8558.060628] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8558.072688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8558.086791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8558.090038] Lustre: Skipped 1 previous similar message [ 8558.679405] Lustre: server umount lustre-MDT0000 complete [ 8561.012977] LustreError: 451563:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761129777 with bad export cookie 10660429053597429812 [ 8561.017227] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8561.020287] LustreError: 451563:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8561.219541] Lustre: server umount lustre-MDT0001 complete [ 8568.567914] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8570.093641] Key type lgssc unregistered [ 8570.249434] LNet: 455247:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8570.253432] LNet: Removed LNI 192.168.204.133@tcp [ 8570.726276] Key type .llcrypt unregistered [ 8570.727837] Key type ._llcrypt unregistered [ 8582.491325] alg: No test for adler32 (adler32-zlib) [ 8583.246430] Key type ._llcrypt registered [ 8583.248172] Key type .llcrypt registered [ 8583.311674] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8583.881095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8584.075887] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 8584.182648] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8584.185996] LNet: Accept secure, port 988 [ 8585.799098] Key type lgssc registered [ 8586.311059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8590.327789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8591.558876] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8591.587276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8593.562091] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8596.962712] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8598.152891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8598.348127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8600.478282] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8604.558492] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8607.508061] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8610.830123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8611.016101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8613.213130] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8617.211195] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8622.311758] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 8624.095793] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8624.099503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.105315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8627.683340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8627.684863] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8627.695495] Lustre: Skipped 1 previous similar message [ 8630.048337] Lustre: server umount lustre-MDT0000 complete [ 8631.155695] LustreError: 456685:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.33@tcp arrived at 1761129847 with bad export cookie 7386302271592052484 [ 8631.162886] LustreError: 456685:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8632.042335] LustreError: 456685:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761129848 with bad export cookie 7386302271592051819 [ 8632.049263] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8632.800675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8632.800731] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8632.809946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8632.815092] LustreError: Skipped 1 previous similar message [ 8632.819923] Lustre: Skipped 3 previous similar messages [ 8637.858234] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8637.858529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8637.864161] LustreError: Skipped 1 previous similar message [ 8638.192305] Lustre: server umount lustre-OST0000 complete [ 8641.645332] Lustre: server umount lustre-MDT0001 complete [ 8647.588578] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8649.014828] Key type lgssc unregistered [ 8649.171656] LNet: 460509:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8649.177942] LNet: Removed LNI 192.168.204.133@tcp [ 8649.624658] Key type .llcrypt unregistered [ 8649.630050] Key type ._llcrypt unregistered [ 8662.459186] alg: No test for adler32 (adler32-zlib) [ 8663.211406] Key type ._llcrypt registered [ 8663.213615] Key type .llcrypt registered [ 8663.269544] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8663.696941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8663.982367] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 8664.101989] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8664.104617] LNet: Accept secure, port 988 [ 8665.711093] Key type lgssc registered [ 8666.264876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8670.369330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8671.629446] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8671.670606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8674.003247] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8676.833074] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8679.092787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.308069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8681.488959] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8685.816521] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8689.234379] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8693.289157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8693.465053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8695.719826] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8699.782408] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8701.670943] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [ 8707.041390] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8707.052796] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8707.056035] Lustre: Skipped 1 previous similar message [ 8710.502575] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 8712.161243] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8712.165434] Lustre: Skipped 1 previous similar message [ 8712.604924] Lustre: server umount lustre-MDT0001 complete [ 8717.282154] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8717.282884] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8717.285734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8717.297361] Lustre: Skipped 1 previous similar message [ 8717.300175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8721.449555] Lustre: server umount lustre-OST0000 complete [ 8722.400262] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8728.543266] Lustre: 465021:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761129939/real 1761129939] req@00000000634d575c x1846678525064896/t0(0) o251->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761129945 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8728.631777] Lustre: server umount lustre-MDT0000 complete [ 8735.830378] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8737.156357] Key type lgssc unregistered [ 8737.319100] LNet: 465775:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8737.323901] LNet: Removed LNI 192.168.204.133@tcp [ 8737.716254] Key type .llcrypt unregistered [ 8737.719042] Key type ._llcrypt unregistered [ 8749.911035] alg: No test for adler32 (adler32-zlib) [ 8750.662567] Key type ._llcrypt registered [ 8750.664853] Key type .llcrypt registered [ 8750.727517] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 8757.159372] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8757.628114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8757.882073] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 8757.976727] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8757.979557] LNet: Accept secure, port 988 [ 8759.583127] Key type lgssc registered [ 8760.140313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8765.313101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8768.867628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8772.083800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8775.166705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8779.855715] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8784.349259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8787.664145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8787.716763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8788.845097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8788.859649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8788.904699] Lustre: lustre-MDT0000: new disk, initializing [ 8788.951107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8788.966575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8791.054428] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8795.852634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8799.169639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8799.217326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8799.221594] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8799.336859] Lustre: lustre-OST0000: new disk, initializing [ 8799.341755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8799.344538] Lustre: Skipped 1 previous similar message [ 8799.375646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8801.436664] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8806.570212] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8810.749481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8813.957854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8813.993802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8814.044127] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8814.073707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8814.115791] Lustre: lustre-MDT0001: new disk, initializing [ 8814.143517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8814.147694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8814.170370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8816.040622] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8820.493725] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 8820.591210] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8822.374270] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 8822.471691] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8826.849523] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8826.849705] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8826.850847] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8826.854388] Lustre: Skipped 1 previous similar message [ 8830.605248] Lustre: server umount lustre-MDT0001 complete [ 8831.969133] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8831.975856] LustreError: Skipped 1 previous similar message [ 8837.089482] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8837.095040] LustreError: Skipped 1 previous similar message [ 8842.209591] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8842.216023] LustreError: Skipped 1 previous similar message [ 8847.329248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8847.329494] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8847.335284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8847.342443] Lustre: Skipped 2 previous similar messages [ 8849.772274] Lustre: server umount lustre-OST0000 complete [ 8863.711269] Lustre: 474185:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761130074/real 1761130074] req@0000000053364ac0 x1846678623633408/t0(0) o251->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761130080 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8863.793667] Lustre: server umount lustre-MDT0000 complete [ 8870.614933] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8871.854929] Key type lgssc unregistered [ 8871.982449] LNet: 474940:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8871.986737] LNet: Removed LNI 192.168.204.133@tcp [ 8872.355452] Key type .llcrypt unregistered [ 8872.357836] Key type ._llcrypt unregistered [ 8888.646031] alg: No test for adler32 (adler32-zlib) [ 8889.398390] Key type ._llcrypt registered [ 8889.399829] Key type .llcrypt registered [ 8889.453921] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8889.566097] Key type .llcrypt unregistered [ 8889.567608] Key type ._llcrypt unregistered [ 8897.120659] alg: No test for adler32 (adler32-zlib) [ 8897.872438] Key type ._llcrypt registered [ 8897.874110] Key type .llcrypt registered [ 8897.920312] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8898.404749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8898.582113] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 8898.674144] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8898.676596] LNet: Accept secure, port 988 [ 8900.263218] Key type lgssc registered [ 8900.738972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8904.076187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8906.832401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8909.140603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8911.706349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8911.731420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8912.828059] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8912.840921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8912.883311] Lustre: lustre-MDT0000: new disk, initializing [ 8912.918949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8912.928069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8914.609370] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8919.542702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8919.576355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8919.617104] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8919.637768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8919.640972] Lustre: Skipped 1 previous similar message [ 8919.688268] Lustre: lustre-MDT0001: new disk, initializing [ 8919.730735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8919.750869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8919.755468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8921.371252] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8925.654410] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8927.931054] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8930.532314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8930.568933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8930.572501] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8930.677520] Lustre: lustre-OST0000: new disk, initializing [ 8930.679763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8930.711652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8932.350445] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8936.806832] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8936.959388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8936.963538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8940.512225] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8940.515530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8940.525251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8942.049528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8942.049832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8942.053987] Lustre: Skipped 1 previous similar message [ 8942.056633] Lustre: Skipped 1 previous similar message [ 8945.621106] Lustre: server umount lustre-MDT0000 complete [ 8947.169338] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8947.176637] LustreError: Skipped 2 previous similar messages [ 8947.268072] LustreError: 478424:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761130163 with bad export cookie 16473684894948050913 [ 8947.271128] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8947.272602] LustreError: 478424:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8947.443708] Lustre: server umount lustre-MDT0001 complete [ 8955.359157] Lustre: 481682:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761130165/real 1761130165] req@00000000f88b335d x1846678771480384/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761130171 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8955.447886] Lustre: server umount lustre-OST0000 complete [ 8959.493405] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 06:49:35 (1761130175) [ 8959.903507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8962.579495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8962.792039] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8962.822660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8964.586511] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8968.090494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8968.159910] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8968.248074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8970.068876] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8973.081965] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8975.634110] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8978.550043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8978.590096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8978.692253] Lustre: Found index 0 for lustre-OST0000, updating log [ 8978.769950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8980.723514] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 8984.062927] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8993.130942] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [ 8998.368513] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8998.369092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8998.371521] LustreError: Skipped 1 previous similar message [ 8998.375985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8998.380198] Lustre: Skipped 2 previous similar messages [ 9002.338955] Lustre: server umount lustre-OST0000 complete [ 9004.001047] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9004.004504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9004.015401] Lustre: Skipped 1 previous similar message [ 9004.019206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9008.608940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9008.609085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9016.487638] Lustre: server umount lustre-MDT0000 complete [ 9017.939108] LustreError: 482487:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761130234 with bad export cookie 16473684894948052236 [ 9017.941648] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9017.944355] LustreError: 482487:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9021.673507] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 06:50:37 (1761130237) [ 9023.988931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9026.011146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9028.224597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9030.330107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9037.111396] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 9040.792982] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 9045.061495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9047.920376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9050.605976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9053.207166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9056.256547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9056.292055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9056.398174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9056.414362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9056.458517] Lustre: lustre-MDT0000: new disk, initializing [ 9056.501379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9056.510783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9058.313712] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 9063.612354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9063.643682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9063.690635] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9063.717412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9063.720456] Lustre: Skipped 1 previous similar message [ 9063.763367] Lustre: lustre-MDT0001: new disk, initializing [ 9063.818666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9063.825149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9065.704805] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 9070.208539] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9072.425817] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9075.132855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9075.179766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9075.183701] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9075.323299] Lustre: lustre-OST0000: new disk, initializing [ 9075.326545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9077.284898] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all 8 [ 9082.112532] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9084.050604] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9084.422319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9084.427358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9085.174190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 9087.063761] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9087.158996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9089.505126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9089.506301] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9089.511325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9089.516026] Lustre: Skipped 6 previous similar messages [ 9094.115411] Lustre: server umount lustre-OST0000 complete [ 9094.117246] Lustre: Skipped 1 previous similar message [ 9099.744427] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9099.747954] LustreError: Skipped 1 previous similar message [ 9103.410094] LustreError: 489538:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761130319 with bad export cookie 16473684894948053293 [ 9103.412518] LustreError: 166-1: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9103.414162] LustreError: 489538:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9113.444544] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 9117.558404] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 9122.000029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9125.517023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9128.365413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9131.532158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9132.456228] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9000 sec ========== 06:52:28 (1761130348) [ 9152.028666] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 9153.190144] Key type lgssc unregistered [ 9153.311324] LNet: 498271:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9153.314694] LNet: Removed LNI 192.168.204.133@tcp [ 9153.665119] Key type .llcrypt unregistered [ 9153.666736] Key type ._llcrypt unregistered