[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478289632 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002401] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.006924] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009149] LSM: Security Framework initializing [ 0.010060] Yama: becoming mindful. [ 0.011032] SELinux: Initializing. [ 0.012071] *** VALIDATE selinux *** [ 0.020405] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024732] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028106] *** VALIDATE tmpfs *** [ 0.029398] *** VALIDATE proc *** [ 0.031046] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033231] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036034] Spectre V2 : User space: Vulnerable [ 0.037012] Speculative Store Bypass: Vulnerable [ 0.040424] debug: unmapping init [mem 0xffffffff87459000-0xffffffff87460fff] [ 0.042144] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043653] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044025] ... version: 2 [ 0.045021] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051296] rcu: Hierarchical SRCU implementation. [ 0.053431] smp: Bringing up secondary CPUs ... [ 0.054590] x86: Booting SMP configuration: [ 0.055026] .... node #0, CPUs: #1 #2 #3 [ 0.058223] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144615] node 0 deferred pages initialised in 80ms [ 0.147327] devtmpfs: initialized [ 0.149264] x86/mm: Memory block size: 128MB [ 0.151877] gcov: version magic: 0x41383552 [ 0.155207] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156155] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157000] pinctrl core: initialized pinctrl subsystem [ 0.157000] [ 0.158021] ************************************************************* [ 0.159000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162017] ** ** [ 0.164019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166019] ** ** [ 0.168020] ** This means that this kernel is built to expose internal ** [ 0.171019] ** IOMMU data structures, which may compromise security on ** [ 0.172016] ** your system. ** [ 0.175020] ** ** [ 0.177013] ** If you see this message and you are not debugging the ** [ 0.179020] ** kernel, report this immediately to your vendor! ** [ 0.182021] ** ** [ 0.184018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187037] ************************************************************* [ 0.189847] NET: Registered protocol family 16 [ 0.191505] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196113] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199101] cpuidle: using governor menu [ 0.201499] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203546] PCI: Using configuration type 1 for base access [ 0.206113] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214114] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220059] cryptd: max_cpu_qlen set to 1000 [ 0.225042] ACPI: Added _OSI(Module Device) [ 0.227013] ACPI: Added _OSI(Processor Device) [ 0.228014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.234558] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241402] ACPI: Interpreter enabled [ 0.242068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244015] ACPI: Using IOAPIC for interrupt routing [ 0.245094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249402] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258229] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267283] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.271210] acpiphp: Slot [2] registered [ 0.273178] acpiphp: Slot [3] registered [ 0.274092] acpiphp: Slot [4] registered [ 0.276121] acpiphp: Slot [5] registered [ 0.277163] acpiphp: Slot [6] registered [ 0.279099] acpiphp: Slot [7] registered [ 0.280112] acpiphp: Slot [8] registered [ 0.282114] acpiphp: Slot [9] registered [ 0.283102] acpiphp: Slot [10] registered [ 0.285134] acpiphp: Slot [11] registered [ 0.286107] acpiphp: Slot [12] registered [ 0.288102] acpiphp: Slot [13] registered [ 0.289104] acpiphp: Slot [14] registered [ 0.291095] acpiphp: Slot [15] registered [ 0.292114] acpiphp: Slot [16] registered [ 0.294106] acpiphp: Slot [17] registered [ 0.295105] acpiphp: Slot [18] registered [ 0.297101] acpiphp: Slot [19] registered [ 0.298100] acpiphp: Slot [20] registered [ 0.300100] acpiphp: Slot [21] registered [ 0.301100] acpiphp: Slot [22] registered [ 0.303100] acpiphp: Slot [23] registered [ 0.304140] acpiphp: Slot [24] registered [ 0.306116] acpiphp: Slot [25] registered [ 0.307098] acpiphp: Slot [26] registered [ 0.309105] acpiphp: Slot [27] registered [ 0.310108] acpiphp: Slot [28] registered [ 0.312071] acpiphp: Slot [29] registered [ 0.313117] acpiphp: Slot [30] registered [ 0.315129] acpiphp: Slot [31] registered [ 0.317061] PCI host bridge to bus 0000:00 [ 0.318018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.320017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.322020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.324021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.326019] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.329028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.330116] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334301] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339586] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.349014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.354059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.362019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364286] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367671] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372611] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.393028] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.399025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405457] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.418015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.436016] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.448102] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.460018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.479018] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.487219] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.493019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.497019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.508020] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.516788] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.526022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.534027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.547023] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.558661] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.564019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.569019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.581018] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.592249] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.603022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.609025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.623024] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.636409] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.640616] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.642543] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.645483] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.647268] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.652141] iommu: Default domain type: Passthrough [ 0.653000] SCSI subsystem initialized [ 0.653189] ACPI: bus type USB registered [ 0.655128] usbcore: registered new interface driver usbfs [ 0.657125] usbcore: registered new interface driver hub [ 0.658134] usbcore: registered new device driver usb [ 0.660220] pps_core: LinuxPPS API ver. 1 registered [ 0.662014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.666080] PTP clock support registered [ 0.668059] EDAC MC: Ver: 3.0.0 [ 0.669533] PCI: Using ACPI for IRQ routing [ 0.671964] NetLabel: Initializing [ 0.673017] NetLabel: domain hash size = 128 [ 0.675016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.677105] NetLabel: unlabeled traffic allowed by default [ 0.679164] vgaarb: loaded [ 0.682065] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.684019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.689230] clocksource: Switched to clocksource kvm-clock [ 0.800326] VFS: Disk quotas dquot_6.6.0 [ 0.801882] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.804267] *** VALIDATE ramfs *** [ 0.805425] *** VALIDATE hugetlbfs *** [ 0.806891] pnp: PnP ACPI init [ 0.809866] pnp: PnP ACPI: found 6 devices [ 0.834978] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.838052] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.840116] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.842644] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.844849] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.847365] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.849991] NET: Registered protocol family 2 [ 0.852757] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.857199] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.860883] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.866157] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.869762] TCP: Hash tables configured (established 65536 bind 65536) [ 0.872767] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.876340] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.879532] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.882672] NET: Registered protocol family 1 [ 0.886248] RPC: Registered named UNIX socket transport module. [ 0.888393] RPC: Registered udp transport module. [ 0.890215] RPC: Registered tcp transport module. [ 0.892042] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.894341] NET: Registered protocol family 44 [ 0.895861] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.897839] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.899671] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.902022] PCI: CLS 0 bytes, default 64 [ 0.903714] Unpacking initramfs... [ 2.295615] debug: unmapping init [mem 0xffff89823cc54000-0xffff89823ffbffff] [ 2.299503] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.302058] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.305390] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.827104] Initialise system trusted keyrings [ 2.829072] Key type blacklist registered [ 2.832088] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.841548] zbud: loaded [ 2.844496] *** VALIDATE nfs *** [ 2.846012] *** VALIDATE nfs4 *** [ 2.847577] pstore: using deflate compression [ 2.850655] Platform Keyring initialized [ 2.952300] NET: Registered protocol family 38 [ 2.954288] Key type asymmetric registered [ 2.955845] Asymmetric key parser 'x509' registered [ 2.957984] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.962179] io scheduler mq-deadline registered [ 2.963667] io scheduler kyber registered [ 2.965257] io scheduler bfq registered [ 2.967403] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.970615] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.973722] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.977050] ACPI: Power Button [PWRF] [ 3.067313] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.160483] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.356909] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.451110] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.647679] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.680891] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.717513] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.723182] Non-volatile memory driver v1.3 [ 3.725075] Linux agpgart interface v0.103 [ 3.759042] virtio_blk virtio1: [vda] 67984 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.763042] vda: detected capacity change from 0 to 34807808 [ 3.778268] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.781631] vdb: detected capacity change from 0 to 1073741824 [ 3.797203] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.800247] vdc: detected capacity change from 0 to 2621440000 [ 3.816388] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.819368] vdd: detected capacity change from 0 to 2621440000 [ 3.832745] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.834890] vde: detected capacity change from 0 to 4294967296 [ 3.848104] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.851119] vdf: detected capacity change from 0 to 4294967296 [ 3.860429] libphy: Fixed MDIO Bus: probed [ 3.868514] usbcore: registered new interface driver usbserial_generic [ 3.871319] usbserial: USB Serial support registered for generic [ 3.873552] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.877151] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.878654] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.880707] mousedev: PS/2 mouse device common for all mice [ 3.883593] rtc_cmos 00:05: RTC can wake from S4 [ 3.886753] rtc_cmos 00:05: registered as rtc0 [ 3.887063] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.888718] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.895195] intel_pstate: CPU model not supported [ 3.898416] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.898720] hid: raw HID events driver (C) Jiri Kosina [ 3.903104] usbcore: registered new interface driver usbhid [ 3.904714] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.905322] usbhid: USB HID core driver [ 3.905467] drop_monitor: Initializing network drop monitor service [ 3.911413] Initializing XFRM netlink socket [ 3.913353] NET: Registered protocol family 10 [ 3.916121] Segment Routing with IPv6 [ 3.917307] NET: Registered protocol family 17 [ 3.919179] mpls_gso: MPLS GSO support [ 3.924499] RAS: Correctable Errors collector initialized. [ 3.926212] AVX version of gcm_enc/dec engaged. [ 3.927634] AES CTR mode by8 optimization enabled [ 4.001124] sched_clock: Marking stable (4001101341, 0)->(4915675084, -914573743) [ 4.005322] registered taskstats version 1 [ 4.007294] Loading compiled-in X.509 certificates [ 4.009530] zswap: loaded using pool lzo/zbud [ 4.033212] Key type big_key registered [ 4.046360] Key type encrypted registered [ 4.048214] ima: No TPM chip found, activating TPM-bypass! [ 4.050313] ima: Allocated hash algorithm: sha1 [ 4.051540] ima: No architecture policies found [ 4.053085] evm: Initialising EVM extended attributes: [ 4.054810] evm: security.selinux [ 4.056025] evm: security.ima [ 4.057162] evm: security.capability [ 4.058170] evm: HMAC attrs: 0x1 [ 4.060487] rtc_cmos 00:05: setting system clock to 2025-10-24 13:46:26 UTC (1761313586) [ 4.066748] debug: unmapping init [mem 0xffffffff88403000-0xffffffff885fffff] [ 4.069652] debug: unmapping init [mem 0xffffffff87182000-0xffffffff87458fff] [ 4.079026] Write protecting the kernel read-only data: 28672k [ 4.082043] debug: unmapping init [mem 0xffffffff85803000-0xffffffff859fffff] [ 4.084936] debug: unmapping init [mem 0xffffffff86114000-0xffffffff861fffff] [ 4.121305] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.130216] systemd[1]: Detected virtualization kvm. [ 4.132085] systemd[1]: Detected architecture x86-64. [ 4.134083] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.166899] systemd[1]: No hostname configured. [ 4.168776] systemd[1]: Set hostname to . [ 4.171063] random: systemd: uninitialized urandom read (16 bytes read) [ 4.173682] systemd[1]: Initializing machine ID from random generator. [ 4.316047] random: systemd: uninitialized urandom read (16 bytes read) [ 4.318661] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.324603] random: systemd: uninitialized urandom read (16 bytes read) [ 4.327835] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.334460] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.014900] device-mapper: uevent: version 1.0.3 [ 5.017685] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.816668] random: fast init done [ 5.852227] virtio_net virtio0 ens2: renamed from eth0 [ 5.939899] scsi host0: ata_piix [ 5.955663] scsi host1: ata_piix [ 5.957801] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.960435] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.709859] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.563786] random: crng init done [ 10.564935] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.110276] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.332408] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.595894] SELinux: Disabled at runtime. [ 12.658427] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.665355] systemd[1]: Detected virtualization kvm. [ 12.667215] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.206966] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.210622] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.217029] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.221278] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.224653] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.241581] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.255465] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ 13.407348] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.711615] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 14.061802] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.080813] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.286659] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.296770] EDAC sbridge: Ver: 1.1.2 [ 15.868833] Key type dns_resolver registered [ 16.177485] NFS: Registering the id_resolver key type [ 16.179173] Key type id_resolver registered [ 16.180485] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg323-server login: [ 38.539273] libcfs: loading out-of-tree module taints kernel. [ 38.561151] alg: No test for adler32 (adler32-zlib) [ 39.315463] Key type ._llcrypt registered [ 39.316946] Key type .llcrypt registered [ 39.367355] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 48.760577] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 49.916330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.429058] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 50.970663] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 50.973966] LNet: Accept secure, port 988 [ 52.656128] Key type lgssc registered [ 53.794578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 63.658558] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 65.141052] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 71.583060] hrtimer: interrupt took 6052675 ns [ 71.904439] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.297508] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.579805] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.273540] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 99.734428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.776497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 100.951962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 100.983734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 101.048166] Lustre: lustre-MDT0000: new disk, initializing [ 101.132546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 101.147523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 103.839285] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 112.032743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.091281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 112.136496] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 112.152741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 112.156451] Lustre: Skipped 1 previous similar message [ 112.211489] Lustre: lustre-MDT0001: new disk, initializing [ 112.266543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 112.286955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 112.298564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 114.934525] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 123.997614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.050385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 124.054643] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 124.230942] Lustre: lustre-OST0000: new disk, initializing [ 124.235817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 124.290783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.949068] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 132.116584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 132.131313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 136.373324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.433491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 136.440126] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 136.514446] Lustre: lustre-OST0001: new disk, initializing [ 136.521109] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 136.563971] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 139.418477] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 141.316098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 141.327626] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 147.486768] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 153.875293] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 161.236764] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing check_logdir /tmp/testlogs/ [ 163.971342] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing yml_node [ 166.727376] Lustre: DEBUG MARKER: Client: 2.15.7.10 [ 168.337156] Lustre: DEBUG MARKER: MDS: 2.15.7.10 [ 169.741983] Lustre: DEBUG MARKER: OSS: 2.15.7.10 [ 170.629595] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 09:49:12 EDT 2025 [ 174.922331] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 175.704610] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 189.410307] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 189.413571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 189.421032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 192.482733] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 192.487265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 192.490784] Lustre: Skipped 2 previous similar messages [ 192.506433] Lustre: Skipped 3 previous similar messages [ 197.601520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 202.727956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 202.734043] Lustre: Skipped 6 previous similar messages [ 203.232765] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 203.318686] Lustre: server umount lustre-MDT0000 complete [ 205.777217] LustreError: 7237:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313788 with bad export cookie 17008893997869763137 [ 205.777585] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 205.781524] LustreError: 7237:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 207.841417] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 207.845633] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 207.845908] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 207.856051] LustreError: Skipped 1 previous similar message [ 207.863831] Lustre: Skipped 1 previous similar message [ 210.849051] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 210.856760] LustreError: Skipped 1 previous similar message [ 211.875417] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 211.881701] LustreError: Skipped 1 previous similar message [ 217.059483] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 217.059596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 217.067983] LustreError: Skipped 1 previous similar message [ 217.072792] Lustre: Skipped 6 previous similar messages [ 220.128159] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 220.413625] Lustre: server umount lustre-MDT0001 complete [ 223.563922] Lustre: server umount lustre-OST0000 complete [ 226.843300] Lustre: server umount lustre-OST0001 complete [ 231.541312] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 237.406343] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 244.860498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 249.409273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 253.078337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 257.007690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 261.168760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 261.217284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 261.357644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 261.381412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 261.446275] Lustre: lustre-MDT0000: new disk, initializing [ 261.527782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 261.541603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 264.046369] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 271.916482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 271.963753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 272.023404] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 272.050166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 272.054159] Lustre: Skipped 1 previous similar message [ 272.132135] Lustre: lustre-MDT0001: new disk, initializing [ 272.213236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.243355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 272.254577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 275.012295] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 283.297883] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 287.771952] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 292.399661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 292.458570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 292.462440] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 292.635552] Lustre: lustre-OST0000: new disk, initializing [ 292.639917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 292.714845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 294.498065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 294.505849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 295.730979] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 302.471462] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 305.400943] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 305.627903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 308.891433] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 309.044958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 315.361887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.363209] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 315.365644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 315.369038] Lustre: Skipped 1 previous similar message [ 316.641807] Lustre: server umount lustre-OST0000 complete [ 320.483232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.478588] Lustre: server umount lustre-MDT0000 complete [ 325.604956] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.612089] LustreError: Skipped 1 previous similar message [ 327.803402] LustreError: 14300:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313910 with bad export cookie 17008893997869764586 [ 327.808328] LustreError: 14300:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 327.811680] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 330.733694] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:51:52 (1761313912) [ 332.517514] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 333.809401] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:51:55 (1761313915) [ 338.219201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.483966] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.522968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 341.403454] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 347.495469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.779159] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 356.281956] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 359.860253] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 363.932797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 367.290791] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 372.915821] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 382.951793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.956981] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 382.957205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 382.957210] Lustre: Skipped 3 previous similar messages [ 382.966578] Lustre: Skipped 2 previous similar messages [ 382.971090] LustreError: Skipped 1 previous similar message [ 383.886178] Lustre: server umount lustre-OST0000 complete [ 383.891934] Lustre: Skipped 1 previous similar message [ 401.376113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 404.319403] LustreError: 18835:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313986 with bad export cookie 17008893997869765447 [ 404.320223] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 404.325678] LustreError: 18835:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 414.559166] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 416.916375] Key type lgssc unregistered [ 417.125815] LNet: 22507:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.132790] LNet: Removed LNI 192.168.203.123@tcp [ 417.782234] Key type .llcrypt unregistered [ 417.784121] Key type ._llcrypt unregistered [ 435.216410] alg: No test for adler32 (adler32-zlib) [ 435.968388] Key type ._llcrypt registered [ 435.972569] Key type .llcrypt registered [ 436.056780] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 436.890560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 437.376418] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 437.548799] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 437.553700] LNet: Accept secure, port 988 [ 439.208113] Key type lgssc registered [ 440.114838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 446.517636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.806253] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 447.836548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.719824] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 453.090863] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.866270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.088352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 460.392479] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 465.794358] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 469.516964] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 473.953064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 474.219188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.218775] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 478.316581] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 482.193085] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 493.029033] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.029874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.036470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.043733] Lustre: Skipped 1 previous similar message [ 494.487132] Lustre: server umount lustre-OST0000 complete [ 498.146035] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 498.149540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.158671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 498.158679] Lustre: Skipped 1 previous similar message [ 498.180355] Lustre: Skipped 1 previous similar message [ 503.265567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 503.268497] Lustre: Skipped 1 previous similar message [ 503.417253] Lustre: server umount lustre-MDT0000 complete [ 505.952213] LustreError: 26613:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314088 with bad export cookie 13883676925589975226 [ 505.953578] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 505.957278] LustreError: 26613:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 506.170669] Lustre: server umount lustre-MDT0001 complete [ 514.538033] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 516.878807] Key type lgssc unregistered [ 517.119548] LNet: 27619:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.125225] LNet: Removed LNI 192.168.203.123@tcp [ 517.776394] Key type .llcrypt unregistered [ 517.777888] Key type ._llcrypt unregistered [ 533.229551] alg: No test for adler32 (adler32-zlib) [ 533.982528] Key type ._llcrypt registered [ 533.984629] Key type .llcrypt registered [ 534.053343] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 534.750141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 534.958749] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 535.070510] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 535.072955] LNet: Accept secure, port 988 [ 536.704153] Key type lgssc registered [ 537.480620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 543.010109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.336035] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.378766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.377788] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 549.862138] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.946830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.186651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 556.851828] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 561.848544] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 565.303191] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 569.481871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 569.689985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.012449] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 574.832884] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 577.576274] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 589.286442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.289429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 589.296853] Lustre: Skipped 1 previous similar message [ 589.310025] Lustre: Skipped 2 previous similar messages [ 592.411801] Lustre: server umount lustre-MDT0000 complete [ 594.402070] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.422821] LustreError: Skipped 2 previous similar messages [ 595.302513] LustreError: 29061:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314177 with bad export cookie 16246018163103439825 [ 595.305852] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.311437] LustreError: 29061:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 599.521781] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.524236] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.533415] LustreError: Skipped 1 previous similar message [ 599.548872] Lustre: Skipped 1 previous similar message [ 599.575438] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 599.578244] Lustre: Skipped 1 previous similar message [ 601.689239] Lustre: server umount lustre-MDT0001 complete [ 630.240165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 630.391845] Lustre: server umount lustre-OST0000 complete [ 643.783726] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 646.238996] Key type lgssc unregistered [ 646.552809] LNet: 33063:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.562693] LNet: Removed LNI 192.168.203.123@tcp [ 647.439754] Key type .llcrypt unregistered [ 647.446803] Key type ._llcrypt unregistered [ 666.293624] alg: No test for adler32 (adler32-zlib) [ 667.044488] Key type ._llcrypt registered [ 667.046348] Key type .llcrypt registered [ 667.162961] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 668.058261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 668.416682] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 668.622667] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 668.627653] LNet: Accept secure, port 988 [ 670.296285] Key type lgssc registered [ 671.291951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 678.030866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.324236] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.369859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 682.985166] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 684.515353] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.625126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.636553] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.880855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 693.263782] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 699.365058] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 703.817319] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 708.899205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 709.201653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 710.262546] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 712.226661] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 717.981663] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 729.967674] Lustre: lustre-OST0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 731.122381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.123229] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 731.131565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 731.134087] Lustre: Skipped 1 previous similar message [ 731.142757] LustreError: Skipped 1 previous similar message [ 732.741462] Lustre: server umount lustre-OST0000 complete [ 751.586157] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 751.602814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 751.635851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 751.637763] Lustre: Skipped 1 previous similar message [ 752.612724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.696377] Lustre: server umount lustre-MDT0000 complete [ 757.729641] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 757.749964] LustreError: Skipped 1 previous similar message [ 758.813707] LustreError: 34508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314341 with bad export cookie 5394368637810403337 [ 758.818375] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 758.825783] LustreError: 34508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 765.202139] Lustre: server umount lustre-MDT0001 complete [ 774.404761] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 776.412829] Key type lgssc unregistered [ 776.611151] LNet: 38356:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.620990] LNet: Removed LNI 192.168.203.123@tcp [ 777.115021] Key type .llcrypt unregistered [ 777.117699] Key type ._llcrypt unregistered [ 793.148466] alg: No test for adler32 (adler32-zlib) [ 793.900335] Key type ._llcrypt registered [ 793.901840] Key type .llcrypt registered [ 793.992889] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 794.978755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 795.270760] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 795.449476] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 795.454420] LNet: Accept secure, port 988 [ 797.096471] Key type lgssc registered [ 797.933580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 805.581627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.001374] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.044956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.111307] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 812.520204] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 817.642392] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 818.754211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.254057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 822.724192] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 828.592420] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 832.356122] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 836.666933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 836.884960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 840.118736] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 845.738991] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 857.327389] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 864.775855] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 867.024886] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 873.454368] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 873.465691] LNetError: 39121:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 873.469378] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 878.561323] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-2410h00m59s [ 879.865639] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 882.029692] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 883.171167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.172162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 883.176903] Lustre: Skipped 2 previous similar messages [ 883.181606] Lustre: Skipped 1 previous similar message [ 883.188106] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 883.191981] LNetError: 39118:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 883.197471] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 887.139127] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 887.147135] Lustre: Skipped 1 previous similar message [ 887.429182] Lustre: server umount lustre-MDT0000 complete [ 888.296056] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 888.306039] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 888.320065] LustreError: Skipped 2 previous similar messages [ 888.327895] LNetError: 39119:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 888.327937] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 890.482236] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 893.409813] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.413630] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 893.419029] LustreError: Skipped 3 previous similar messages [ 893.424735] LNetError: 39120:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 893.433166] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 895.328136] Lustre: 39131:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314470/real 1761314470] req@00000000b0fb3a25 x1846871984196672/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314477 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 895.364146] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 896.003621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 900.590250] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xd9408c4e2cb01beb to 0xd9408c4e2cb02052 [ 900.603882] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 900.876794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.930060] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 904.297857] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 906.224355] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 906.239104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 906.242863] Lustre: Skipped 1 previous similar message [ 907.143162] Lustre: DEBUG MARKER: Wait for df (28865) ... [ 908.278715] Lustre: DEBUG MARKER: done [ 910.819637] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 910.826063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.833834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.333966] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 916.450843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 916.457652] Lustre: Skipped 2 previous similar messages [ 916.642575] Lustre: server umount lustre-OST0000 complete [ 921.569542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.586715] Lustre: Skipped 2 previous similar messages [ 921.603112] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 921.613505] LNetError: 39121:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 921.619259] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 925.436784] Lustre: server umount lustre-MDT0000 complete [ 926.689106] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.697338] LustreError: Skipped 8 previous similar messages [ 928.307668] LustreError: 42539:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314510 with bad export cookie 15654666572124921938 [ 928.308253] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 928.315349] LustreError: 42539:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 928.581339] Lustre: server umount lustre-MDT0001 complete [ 938.277477] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 940.589849] Key type lgssc unregistered [ 940.837604] LNet: 44955:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.850065] LNet: Removed LNI 192.168.203.123@tcp [ 941.566536] Key type .llcrypt unregistered [ 941.568754] Key type ._llcrypt unregistered [ 968.876591] alg: No test for adler32 (adler32-zlib) [ 969.628424] Key type ._llcrypt registered [ 969.629962] Key type .llcrypt registered [ 969.734638] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 982.619385] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 983.677749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.959234] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 984.181544] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 984.183893] LNet: Accept secure, port 988 [ 985.800124] Key type lgssc registered [ 986.660911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 994.833375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1000.975627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1006.036051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1011.283914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1016.018799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1016.079099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.245894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1017.272958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1017.329319] Lustre: lustre-MDT0000: new disk, initializing [ 1017.445232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.483353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1020.554404] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1028.403185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1028.447491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.528439] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1028.567987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1028.571067] Lustre: Skipped 1 previous similar message [ 1028.644249] Lustre: lustre-MDT0001: new disk, initializing [ 1028.716463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1028.754114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1028.761562] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1031.681327] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1039.463772] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1043.730041] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1048.653726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1048.713504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1048.718890] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1048.929709] Lustre: lustre-OST0000: new disk, initializing [ 1048.938202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1049.047859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.751470] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1058.380812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1058.393328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1060.544427] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1064.309774] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1064.503610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1067.732304] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1067.890462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1074.150730] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1074.150869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.151610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1074.171876] Lustre: Skipped 1 previous similar message [ 1075.397262] Lustre: server umount lustre-OST0000 complete [ 1079.266531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.288309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1079.296582] Lustre: Skipped 2 previous similar messages [ 1084.385778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.389402] Lustre: Skipped 1 previous similar message [ 1089.505759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1089.512402] Lustre: Skipped 1 previous similar message [ 1092.064148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1092.157653] Lustre: server umount lustre-MDT0000 complete [ 1094.864027] LustreError: 49057:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314677 with bad export cookie 530219680728036457 [ 1094.865686] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1094.870665] LustreError: 49057:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1095.089781] Lustre: server umount lustre-MDT0001 complete [ 1101.437229] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 10:04:43 (1761314683) [ 1106.769721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.186515] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1107.272359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.212543] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1112.546521] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.665642] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.837588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.135301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1121.860087] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1127.746910] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1132.208097] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1136.988129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1137.219089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.787789] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1146.767950] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1155.625326] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-2410h05m36s [ 1157.030953] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1162.489468] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1164.115557] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1164.208521] LustreError: 53607:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1165.296462] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 1165.302604] LNetError: 47276:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1165.308575] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1184.665751] Lustre: lustre-MDT0000: Client 12c366cd-ad6d-43a1-b43b-43ea9ad95b2e (at 192.168.203.23@tcp) reconnecting [ 1204.224237] LustreError: 53607:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1205.839277] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1207.761813] Lustre: DEBUG MARKER: done [ 1214.945164] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1214.949144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.957441] LustreError: Skipped 1 previous similar message [ 1214.963211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1214.964343] Lustre: Skipped 2 previous similar messages [ 1214.992880] Lustre: Skipped 3 previous similar messages [ 1218.006500] Lustre: server umount lustre-OST0000 complete [ 1225.192650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.213183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.223237] Lustre: Skipped 1 previous similar message [ 1225.238692] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 1225.250359] LNetError: 47273:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1225.260323] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1227.473166] Lustre: server umount lustre-MDT0000 complete [ 1230.306779] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.313823] LustreError: Skipped 1 previous similar message [ 1230.324198] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 1230.338918] LNetError: 47274:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1230.341556] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1230.506112] LustreError: 56403:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314812 with bad export cookie 530219680728037311 [ 1230.508386] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.519441] LustreError: 56403:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1240.009710] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1242.195146] Key type lgssc unregistered [ 1242.388558] LNet: 58020:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.404614] LNet: Removed LNI 192.168.203.123@tcp [ 1243.118579] Key type .llcrypt unregistered [ 1243.120403] Key type ._llcrypt unregistered [ 1270.423935] alg: No test for adler32 (adler32-zlib) [ 1271.175127] Key type ._llcrypt registered [ 1271.176850] Key type .llcrypt registered [ 1271.273254] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1280.798924] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1281.493130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1281.740354] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 1281.884559] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1281.886955] LNet: Accept secure, port 988 [ 1283.512800] Key type lgssc registered [ 1284.231416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1291.213332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1297.708703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1302.914731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1308.004483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1313.369599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1313.414865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.605931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1314.633353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1314.685653] Lustre: lustre-MDT0000: new disk, initializing [ 1314.778464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.797996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1317.629731] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1325.854760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1325.895937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.975599] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1326.025125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1326.027460] Lustre: Skipped 1 previous similar message [ 1326.083819] Lustre: lustre-MDT0001: new disk, initializing [ 1326.143227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1326.170644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1326.175668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1328.913124] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1335.590975] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1340.211417] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1345.295688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1345.344172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1345.349194] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1345.501667] Lustre: lustre-OST0000: new disk, initializing [ 1345.506861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1345.566832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1348.707758] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1353.272796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1353.279788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1356.009074] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1359.229791] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1359.375433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1362.554924] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1362.701664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1369.057729] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1369.058185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.059348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1369.062857] LustreError: Skipped 1 previous similar message [ 1370.253403] Lustre: server umount lustre-OST0000 complete [ 1374.179540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.181353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1374.187102] Lustre: Skipped 2 previous similar messages [ 1374.191691] Lustre: Skipped 1 previous similar message [ 1379.049260] Lustre: server umount lustre-MDT0000 complete [ 1379.297073] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1379.308044] LustreError: Skipped 1 previous similar message [ 1381.297425] LustreError: 62112:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314963 with bad export cookie 3245567526303486243 [ 1381.298590] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1381.309648] LustreError: 62112:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1381.504558] Lustre: server umount lustre-MDT0001 complete [ 1387.554270] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 10:09:29 (1761314969) [ 1388.349229] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1389.351987] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 10:09:31 (1761314971) [ 1395.835220] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1398.008753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1398.256375] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.300473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.948685] Lustre: Failing over lustre-MDT0000 [ 1398.952301] LustreError: 67132:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1398.957349] LustreError: 67132:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1398.962874] LustreError: 67132:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1405.409347] Lustre: 67183:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314981/real 1761314981] req@000000005c7a54ac x1846872494853504/t0(0) o251->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314987 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1405.486473] Lustre: server umount lustre-MDT0000 complete [ 1408.955795] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:09:50 (1761314990) [ 1412.519927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.773333] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.801439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1415.574654] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1418.222223] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.140324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.386744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1424.031588] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1429.222728] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1433.269466] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1438.668985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1438.941785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.037558] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1447.993788] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1456.281883] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1462.240886] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1462.250979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.270630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1463.570915] Lustre: server umount lustre-MDT0000 complete [ 1466.385430] LustreError: 67731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315048 with bad export cookie 3245567526303487370 [ 1466.393773] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1469.725582] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1472.669451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1475.041462] Lustre: 60340:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315050/real 1761315050] req@00000000095ea192 x1846872494870208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315057 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1475.064712] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.081962] Lustre: Skipped 2 previous similar messages [ 1479.137728] Lustre: 60339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315054/real 1761315054] req@00000000d918d3c2 x1846872494870272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315061 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1479.684835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.309003] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0x2d0a92782bdcd58a to 0x2d0a92782bdcdd16 [ 1483.329422] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 1483.736673] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1483.857372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.729574] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1491.942247] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1491.953327] LustreError: Skipped 2 previous similar messages [ 1495.032530] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1495.078864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1495.484550] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1498.796211] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1500.647920] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1504.976452] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1509.252290] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1516.005329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1516.006435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1516.009766] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1516.020694] Lustre: Skipped 2 previous similar messages [ 1516.039295] Lustre: Skipped 4 previous similar messages [ 1516.169432] Lustre: server umount lustre-MDT0000 complete [ 1516.170962] Lustre: Skipped 1 previous similar message [ 1519.010914] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1519.025713] LustreError: Skipped 6 previous similar messages [ 1519.306362] LustreError: 67731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315101 with bad export cookie 3245567526303489302 [ 1519.311039] LustreError: 67731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1519.313510] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1524.371357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1530.995945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.360316] Lustre: 60341:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315106/real 1761315106] req@000000004be05226 x1846872494884032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315113 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1531.376053] Lustre: 60341:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1531.382497] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1536.480745] Lustre: 60338:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315111/real 1761315111] req@00000000e9595832 x1846872494884224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315118 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1536.501650] Lustre: 60338:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1536.546553] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0x2d0a92782bdcdd16 to 0x2d0a92782bdce328 [ 1536.555613] Lustre: MGC192.168.203.123@tcp: Connection restored to (at 0@lo) [ 1536.884468] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1539.781913] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1545.201234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1545.437878] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1548.080721] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1550.829206] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1550.834475] Lustre: Skipped 1 previous similar message [ 1553.746353] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1558.164119] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1566.178595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.180604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.183717] Lustre: Skipped 2 previous similar messages [ 1566.186705] Lustre: Skipped 1 previous similar message [ 1566.188921] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1568.365992] Lustre: server umount lustre-MDT0000 complete [ 1568.369548] Lustre: Skipped 1 previous similar message [ 1570.694558] LustreError: 70705:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315153 with bad export cookie 3245567526303490856 [ 1570.696314] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1570.704186] LustreError: 70705:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1574.978522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1577.245956] Lustre: DEBUG MARKER: files compared the same [ 1578.383814] Lustre: 60339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315153/real 1761315153] req@000000003c7f587c x1846872494895616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315160 ref 1 fl Rpc:EXNQr/0/ffffffff rc -5/-1 job:'' [ 1578.394828] Lustre: 60339:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1594.363279] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1596.242642] Key type lgssc unregistered [ 1596.457783] LNet: 77788:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.464206] LNet: Removed LNI 192.168.203.123@tcp [ 1597.081420] Key type .llcrypt unregistered [ 1597.084188] Key type ._llcrypt unregistered [ 1611.911620] alg: No test for adler32 (adler32-zlib) [ 1612.664392] Key type ._llcrypt registered [ 1612.666266] Key type .llcrypt registered [ 1612.732886] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1613.275717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1613.647930] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 1613.805773] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1613.809662] LNet: Accept secure, port 988 [ 1615.440479] Key type lgssc registered [ 1616.484842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1622.595216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.968322] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1624.024548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1628.062320] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all 8 [ 1629.153798] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.278945] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.693429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1635.010620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1638.547458] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all 8 [ 1643.967318] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1647.076255] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1650.913744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1651.089779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.193814] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1654.428818] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all 8 [ 1659.770392] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1665.445329] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1669.601720] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1669.605990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.613326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1670.626029] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1670.627661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1670.628399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1670.647898] Lustre: Skipped 1 previous similar message [ 1674.942901] Lustre: server umount lustre-OST0000 complete [ 1680.866956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.868041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1680.872441] Lustre: Skipped 1 previous similar message [ 1683.868148] Lustre: server umount lustre-MDT0000 complete [ 1685.990191] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1685.999052] LustreError: Skipped 1 previous similar message [ 1686.968111] LustreError: 79231:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315269 with bad export cookie 17543566269901845293 [ 1686.984131] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1687.206779] Lustre: server umount lustre-MDT0001 complete [ 1694.891080] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1696.955913] Key type lgssc unregistered [ 1697.171406] LNet: 82902:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.178264] LNet: Removed LNI 192.168.203.123@tcp [ 1697.779685] Key type .llcrypt unregistered [ 1697.781594] Key type ._llcrypt unregistered [ 1715.046731] alg: No test for adler32 (adler32-zlib) [ 1715.806061] Key type ._llcrypt registered [ 1715.810491] Key type .llcrypt registered [ 1715.931795] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1716.570892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1716.971897] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 1717.144970] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1717.150871] LNet: Accept secure, port 988 [ 1718.832120] Key type lgssc registered [ 1719.708279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1725.669259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1742.241239] LustreError: 84336:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 1: rc = -5 [ 1773.984282] LustreError: 84336:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 1803.744195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.814327] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1813.828731] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1820.311572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.564575] Lustre: *** cfs_fail_loc=706, val=0*** [ 1826.566901] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.625546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.696296] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1829.603183] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1831.906080] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.028340] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.297070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.593993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1837.643259] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1841.174787] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1848.221302] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1852.222207] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1857.506666] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1857.521208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.546933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1858.020277] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1858.024670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.043101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.048461] Lustre: Skipped 2 previous similar messages [ 1864.163423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.166170] Lustre: Skipped 1 previous similar message [ 1868.256319] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1868.369609] Lustre: server umount lustre-OST0000 complete [ 1873.380027] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1873.386874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.398080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.520932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.527820] Lustre: Skipped 3 previous similar messages [ 1885.153087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1885.257418] Lustre: server umount lustre-MDT0000 complete [ 1888.325795] LustreError: 87272:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315470 with bad export cookie 8575096870446298244 [ 1888.329418] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.342367] LustreError: 87272:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1888.581355] Lustre: server umount lustre-MDT0001 complete [ 1897.447955] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1900.123690] Key type lgssc unregistered [ 1900.337959] LNet: 88086:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.345642] LNet: Removed LNI 192.168.203.123@tcp [ 1901.158959] Key type .llcrypt unregistered [ 1901.160674] Key type ._llcrypt unregistered [ 1919.485892] alg: No test for adler32 (adler32-zlib) [ 1920.238688] Key type ._llcrypt registered [ 1920.240969] Key type .llcrypt registered [ 1920.314848] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1921.036185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1921.337260] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 1921.488818] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1921.491463] LNet: Accept secure, port 988 [ 1923.137555] Key type lgssc registered [ 1923.817769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1928.750555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.096691] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1937.358911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.614012] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.711317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1940.758723] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1943.014606] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.020280] LustreError: Skipped 2 previous similar messages [ 1947.000107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1947.221135] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.233820] LustreError: Skipped 2 previous similar messages [ 1947.252454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.118432] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1952.740589] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.754968] LustreError: Skipped 2 previous similar messages [ 1953.781651] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 1957.640064] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1957.652804] LustreError: Skipped 5 previous similar messages [ 1957.816201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.864776] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 1960.650533] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1973.218675] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1973.221343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.246401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.249829] Lustre: Skipped 1 previous similar message [ 1976.732045] Lustre: server umount lustre-OST0000 complete [ 1983.463897] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1983.465419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.480889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.497878] Lustre: Skipped 1 previous similar message [ 1983.502461] Lustre: Skipped 1 previous similar message [ 1986.337782] Lustre: server umount lustre-MDT0000 complete [ 1988.578403] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.585688] LustreError: Skipped 1 previous similar message [ 1989.842992] Lustre: server umount lustre-MDT0001 complete [ 1999.328285] Lustre: 92950:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315575/real 1761315575] req@00000000ebccc902 x1846873164886336/t0(0) o251->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315581 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1999.383290] Lustre: server umount MGS complete [ 2006.844950] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:19:48 (1761315588) [ 2026.998636] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2033.420578] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2042.844696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2047.958748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2052.418685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2056.956120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2062.342786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2062.400188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.642402] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2072.869938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2072.910743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.046809] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2073.088911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2073.140939] Lustre: lustre-MDT0001: new disk, initializing [ 2073.203130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2073.222464] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2073.229314] LustreError: 97649:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 2076.694534] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2085.245248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2085.295575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2085.303591] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2085.484063] Lustre: lustre-OST0000: new disk, initializing [ 2085.487525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2085.539664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.357242] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2096.521815] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2096.673860] Lustre: 99088:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@0000000012778ce6 x1846873164891008/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761315691 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2102.530625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2102.581249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2102.658280] Lustre: lustre-MDT0000: new disk, initializing [ 2102.769116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.806281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2103.842260] Lustre: 99499:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000bb2cfc10 x1846873164894272/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1761315698 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2105.917152] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2107.302027] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2108.974646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2108.983408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2110.957127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2110.963025] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2131.425043] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2131.429690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.436300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2131.438577] Lustre: Skipped 1 previous similar message [ 2133.474894] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2133.485186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.496463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2135.750494] Lustre: server umount lustre-OST0000 complete [ 2139.105846] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2139.110722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.111133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.111136] Lustre: Skipped 1 previous similar message [ 2144.643865] Lustre: server umount lustre-MDT0000 complete [ 2163.680157] Lustre: 100694:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315740/real 1761315740] req@000000002799398d x1846873164903680/t0(0) o251->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315746 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2163.730644] Lustre: server umount MGS complete [ 2163.731967] Lustre: Skipped 1 previous similar message [ 2171.328669] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:22:33 (1761315753) [ 2184.782257] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2187.386653] Key type lgssc unregistered [ 2187.674189] LNet: 102099:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.687611] LNet: Removed LNI 192.168.203.123@tcp [ 2188.329949] Key type .llcrypt unregistered [ 2188.331525] Key type ._llcrypt unregistered [ 2201.290203] alg: No test for adler32 (adler32-zlib) [ 2202.043562] Key type ._llcrypt registered [ 2202.045547] Key type .llcrypt registered [ 2202.111737] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2202.881044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2203.139579] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2203.315461] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2203.319957] LNet: Accept secure, port 988 [ 2205.003339] Key type lgssc registered [ 2205.607983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2211.976678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.338753] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2213.384780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.655538] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2218.470817] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.606890] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.766788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.986523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2228.182552] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2234.704235] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2239.588877] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2245.678416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2246.006129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.501615] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2254.985879] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2263.404786] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2273.764515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.767131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2273.786962] Lustre: Skipped 1 previous similar message [ 2277.492645] Lustre: server umount lustre-OST0000 complete [ 2281.441038] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2281.451810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.472862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2281.480790] Lustre: Skipped 1 previous similar message [ 2284.001723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.011159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.122254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.125177] Lustre: Skipped 1 previous similar message [ 2295.264466] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2295.268668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.287545] Lustre: Skipped 3 previous similar messages [ 2295.405769] Lustre: server umount lustre-MDT0000 complete [ 2299.396993] LustreError: 103349:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315881 with bad export cookie 9278091555470662463 [ 2299.399777] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2299.416349] LustreError: 103349:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2299.975054] Lustre: server umount lustre-MDT0001 complete [ 2308.612303] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2310.341616] Key type lgssc unregistered [ 2310.510062] LNet: 107046:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2310.517169] LNet: Removed LNI 192.168.203.123@tcp [ 2311.054521] Key type .llcrypt unregistered [ 2311.056978] Key type ._llcrypt unregistered [ 2322.576922] alg: No test for adler32 (adler32-zlib) [ 2323.329416] Key type ._llcrypt registered [ 2323.332112] Key type .llcrypt registered [ 2323.410327] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2324.076053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2324.329694] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2324.451302] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2324.456765] LNet: Accept secure, port 988 [ 2326.064148] Key type lgssc registered [ 2326.987753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2332.228107] LustreError: 108175:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2332.231663] LustreError: 108173:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2332.236691] LustreError: 108175:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4996 [ 2332.243351] LustreError: 108173:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2332.252366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.268534] LustreError: 108173:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2333.604724] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.656451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.786139] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.033903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.221694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2342.242285] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2346.272907] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing lsmod [ 2348.257359] LustreError: 109481:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2348.264538] LustreError: 109482:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2348.273657] LustreError: 109481:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4990 [ 2348.281492] LustreError: 109481:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2348.289436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2348.293098] LustreError: 109481:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2348.616469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.680278] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2353.764367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.770973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2353.775597] Lustre: Skipped 1 previous similar message [ 2353.796470] Lustre: Skipped 2 previous similar messages [ 2356.749757] Lustre: server umount lustre-MDT0000 complete [ 2359.267759] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.280074] LustreError: Skipped 2 previous similar messages [ 2359.796202] LustreError: 108191:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315942 with bad export cookie 14678537769947154300 [ 2359.800429] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2359.806210] LustreError: 108191:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2360.099191] Lustre: server umount lustre-MDT0001 complete [ 2369.504176] Lustre: 110075:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315945/real 1761315945] req@000000000e357326 x1846873587459648/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315951 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2369.730537] Lustre: server umount lustre-OST0000 complete [ 2377.021610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.457638] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.546688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2380.798828] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2382.821769] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.410875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.701355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2391.072357] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2397.692927] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2402.525085] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2407.984700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2408.280863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.335833] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 2411.953688] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2418.535789] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2423.777876] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2423.782545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.792156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2423.799299] Lustre: Skipped 2 previous similar messages [ 2425.078757] Lustre: server umount lustre-OST0000 complete [ 2428.896878] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2428.898657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.903390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2428.917563] Lustre: Skipped 1 previous similar message [ 2428.923820] Lustre: Skipped 1 previous similar message [ 2434.019533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2434.031795] Lustre: Skipped 2 previous similar messages [ 2434.658435] Lustre: server umount lustre-MDT0000 complete [ 2437.762500] LustreError: 110531:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316020 with bad export cookie 14678537769947155231 [ 2437.763670] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2437.766933] LustreError: 110531:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2448.120663] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2450.192179] Key type lgssc unregistered [ 2450.394327] LNet: 114204:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2450.404229] LNet: Removed LNI 192.168.203.123@tcp [ 2451.134134] Key type .llcrypt unregistered [ 2451.136451] Key type ._llcrypt unregistered [ 2471.067954] alg: No test for adler32 (adler32-zlib) [ 2471.819585] Key type ._llcrypt registered [ 2471.820961] Key type .llcrypt registered [ 2471.920217] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2472.537433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2473.053323] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2473.279449] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2473.283390] LNet: Accept secure, port 988 [ 2474.920164] Key type lgssc registered [ 2475.614840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2480.623306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.950745] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.999507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.251248] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2487.265951] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.387764] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.865139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.255049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2497.070633] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2503.344980] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2508.165285] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2514.166894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2514.491911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.926762] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2523.799540] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2531.893208] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2537.692852] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2540.144377] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2545.491768] Lustre: server umount lustre-OST0000 complete [ 2549.729401] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2549.738869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.745664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2552.305392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2552.310166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.824185] Lustre: 118640:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316130/real 1761316130] req@000000001d96f744 x1846873743705088/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761316136 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2553.826171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.854990] Lustre: 118640:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2553.873421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.884841] Lustre: Skipped 2 previous similar messages [ 2553.999601] Lustre: server umount lustre-MDT0000 complete [ 2555.362095] Lustre: 114979:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316130/real 1761316130] req@00000000add83cbe x1846873743704576/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761316137 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2556.762615] LustreError: 115648:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316139 with bad export cookie 298010436296035652 [ 2556.764909] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2556.776913] LustreError: 115648:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2557.045513] Lustre: server umount lustre-MDT0001 complete [ 2565.109367] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2566.763465] Key type lgssc unregistered [ 2566.919746] LNet: 119443:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2566.924099] LNet: Removed LNI 192.168.203.123@tcp [ 2567.371319] Key type .llcrypt unregistered [ 2567.372956] Key type ._llcrypt unregistered [ 2577.266060] alg: No test for adler32 (adler32-zlib) [ 2578.018431] Key type ._llcrypt registered [ 2578.020063] Key type .llcrypt registered [ 2578.120741] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2578.523897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2578.826220] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2578.950518] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2578.953984] LNet: Accept secure, port 988 [ 2580.576199] Key type lgssc registered [ 2581.198565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2586.817153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.124588] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.162776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.322358] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2593.252431] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.214857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.369847] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2598.527762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2601.500557] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2607.047517] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2611.559324] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2616.436507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.703598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.709448] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 2619.370340] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2623.893363] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2633.187387] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2633.194902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.204036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2633.209670] Lustre: Skipped 1 previous similar message [ 2638.307161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2638.309704] Lustre: Skipped 1 previous similar message [ 2638.827458] Lustre: server umount lustre-OST0000 complete [ 2643.427042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2643.435452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2643.441193] Lustre: Skipped 2 previous similar messages [ 2643.450411] Lustre: Skipped 1 previous similar message [ 2648.550129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2648.556900] Lustre: Skipped 1 previous similar message [ 2656.230209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2656.320304] Lustre: server umount lustre-MDT0000 complete [ 2658.785145] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.791428] LustreError: Skipped 1 previous similar message [ 2659.253973] LustreError: 120689:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316241 with bad export cookie 11682656308566410799 [ 2659.257657] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2659.263325] LustreError: 120689:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2659.589549] Lustre: server umount lustre-MDT0001 complete [ 2668.265965] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2669.976701] Key type lgssc unregistered [ 2670.184476] LNet: 124364:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2670.194338] LNet: Removed LNI 192.168.203.123@tcp [ 2670.867591] Key type .llcrypt unregistered [ 2670.869258] Key type ._llcrypt unregistered [ 2688.681795] alg: No test for adler32 (adler32-zlib) [ 2689.435538] Key type ._llcrypt registered [ 2689.437613] Key type .llcrypt registered [ 2689.563754] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2690.202083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2690.583978] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2690.761892] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2690.764272] LNet: Accept secure, port 988 [ 2692.376261] Key type lgssc registered [ 2693.099321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2698.061392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.372733] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.439518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.224476] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2704.869442] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.225632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.485905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2711.260320] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2715.481707] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2718.543445] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2722.439677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2722.693590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.107864] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2729.839733] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 2732.849582] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2739.191298] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2742.393813] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2742.399626] Lustre: Skipped 2 previous similar messages [ 2746.029388] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2746.037134] Lustre: Skipped 2 previous similar messages [ 2746.927716] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2746.965181] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2755.066366] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2755.070870] Lustre: Skipped 2 previous similar messages [ 2760.788225] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.23@tcp 192.168.0.[2,10]@tcp [ 2764.771310] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2764.777926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2764.790799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.798911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.809428] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2765.813449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2769.834595] Lustre: server umount lustre-OST0000 complete [ 2775.008878] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2775.017668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.029815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.032887] Lustre: Skipped 1 previous similar message [ 2778.696821] Lustre: server umount lustre-MDT0000 complete [ 2781.154588] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.162721] LustreError: Skipped 1 previous similar message [ 2781.540453] LustreError: 125806:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316363 with bad export cookie 584261645346318760 [ 2781.542919] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2781.560994] LustreError: 125806:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2781.861746] Lustre: server umount lustre-MDT0001 complete [ 2791.803489] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2794.171313] Key type lgssc unregistered [ 2794.379607] LNet: 130479:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2794.386798] LNet: Removed LNI 192.168.203.123@tcp [ 2795.089356] Key type .llcrypt unregistered [ 2795.093555] Key type ._llcrypt unregistered [ 2815.480436] alg: No test for adler32 (adler32-zlib) [ 2816.232439] Key type ._llcrypt registered [ 2816.236312] Key type .llcrypt registered [ 2816.347911] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2817.124226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2817.378798] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2817.537301] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2817.539900] LNet: Accept secure, port 988 [ 2819.193196] Key type lgssc registered [ 2819.920632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2825.607180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.938399] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2826.979234] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2827.013683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2827.017446] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.203.23@tcp 192.168.0.[2,10]@tcp [ 2830.216608] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2832.356957] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.489810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.647818] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2836.660279] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.23@tcp 192.168.0.[2,10]@tcp [ 2836.688075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2839.648912] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2845.210663] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2849.678599] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2854.486764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2854.729123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.821755] Lustre: lustre-OST0000: deleting orphan objects from 0x0:199 to 0x0:225 [ 2857.538775] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2863.604722] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2876.898885] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2876.903833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.913738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2877.416292] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2877.423765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.424978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2877.454241] Lustre: Skipped 1 previous similar message [ 2880.442617] Lustre: server umount lustre-OST0000 complete [ 2887.649271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.650227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2887.656478] Lustre: Skipped 1 previous similar message [ 2889.705428] Lustre: server umount lustre-MDT0000 complete [ 2892.048355] LustreError: 132065:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316474 with bad export cookie 5774098871467063596 [ 2892.051721] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2892.057451] LustreError: 132065:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2892.362186] Lustre: server umount lustre-MDT0001 complete [ 2902.695263] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2905.998773] Key type lgssc unregistered [ 2906.389572] LNet: 135795:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2906.412094] LNet: Removed LNI 192.168.203.123@tcp [ 2907.330914] Key type .llcrypt unregistered [ 2907.343481] Key type ._llcrypt unregistered [ 2931.140732] alg: No test for adler32 (adler32-zlib) [ 2931.894521] Key type ._llcrypt registered [ 2931.896211] Key type .llcrypt registered [ 2931.960220] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2941.965805] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2942.870652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2943.083949] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2943.201547] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2943.204669] LNet: Accept secure, port 988 [ 2944.840129] Key type lgssc registered [ 2945.636727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2952.453526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2958.027328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2963.356059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2969.564576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2975.478863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2975.526288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.768345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2976.813408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2976.927475] Lustre: lustre-MDT0000: new disk, initializing [ 2977.056123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2977.073151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2980.704896] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2990.089367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2990.167723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.276130] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2990.328647] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.353685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2990.358228] Lustre: Skipped 1 previous similar message [ 2990.419443] Lustre: lustre-MDT0001: new disk, initializing [ 2990.514025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2990.553803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2990.564551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2993.524617] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3000.214881] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3003.907328] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3008.567748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3008.614411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3008.619920] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3008.770746] Lustre: lustre-OST0000: new disk, initializing [ 3008.777206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3008.843834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3011.556788] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3017.794568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3017.802738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3019.319329] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3022.486948] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3022.661244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3025.452865] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3025.614605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3028.450967] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3028.453074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.454465] LustreError: Skipped 1 previous similar message [ 3028.462549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3028.469067] Lustre: Skipped 1 previous similar message [ 3033.571970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3033.574927] Lustre: Skipped 1 previous similar message [ 3038.690964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3038.693948] Lustre: Skipped 1 previous similar message [ 3041.248844] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3041.372995] Lustre: server umount lustre-OST0000 complete [ 3043.810336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3043.816124] Lustre: Skipped 1 previous similar message [ 3043.816720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3043.820936] Lustre: Skipped 1 previous similar message [ 3048.929866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3048.934619] Lustre: Skipped 1 previous similar message [ 3049.818485] Lustre: server umount lustre-MDT0000 complete [ 3051.940482] LustreError: 140167:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316634 with bad export cookie 1676392690956735748 [ 3051.944359] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3051.946477] LustreError: 140167:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3052.154070] Lustre: server umount lustre-MDT0001 complete [ 3057.280549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.619597] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.687606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.558498] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3062.756024] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.202701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.473464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3069.172404] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3074.681932] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3078.688279] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3083.157595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3083.379885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3086.097445] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3091.237444] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3094.112716] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3094.251217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3096.895664] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3097.038121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3102.253544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3102.299082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3102.307273] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3102.415048] Lustre: lustre-OST0001: new disk, initializing [ 3102.417494] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3102.463921] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3105.133696] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3111.374355] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3111.488530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3111.494149] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3112.626801] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3115.707931] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3115.846913] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3120.083314] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff989342e12800.ost_server_uuid 40 [ 3121.082471] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff989342e12800.ost_server_uuid in FULL state after 0 sec [ 3127.273807] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3127.274500] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3127.275433] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3127.291340] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3127.303569] Lustre: Skipped 1 previous similar message [ 3131.414122] Lustre: server umount lustre-OST0001 complete [ 3132.394281] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.405756] LustreError: Skipped 1 previous similar message [ 3134.946637] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3134.953108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.959930] Lustre: Skipped 1 previous similar message [ 3137.507792] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.522112] LustreError: Skipped 1 previous similar message [ 3140.561275] Lustre: server umount lustre-OST0000 complete [ 3147.746604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.752261] Lustre: Skipped 1 previous similar message [ 3147.756946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3147.760390] Lustre: Skipped 4 previous similar messages [ 3149.448191] Lustre: server umount lustre-MDT0000 complete [ 3152.405252] LustreError: 143656:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316734 with bad export cookie 1676392690956736602 [ 3152.405695] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3152.411741] LustreError: 143656:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3169.402322] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3171.328888] Key type lgssc unregistered [ 3171.514045] LNet: 150330:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3171.519912] LNet: Removed LNI 192.168.203.123@tcp [ 3172.075282] Key type .llcrypt unregistered [ 3172.077358] Key type ._llcrypt unregistered [ 3197.614262] alg: No test for adler32 (adler32-zlib) [ 3198.368128] Key type ._llcrypt registered [ 3198.370669] Key type .llcrypt registered [ 3198.477532] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3209.519890] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3210.397562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3210.927228] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 3211.114187] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3211.124989] LNet: Accept secure, port 988 [ 3212.776159] Key type lgssc registered [ 3213.638809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3221.156026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3226.652291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3231.488700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3236.078096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3240.930092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3240.987941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3242.155358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3242.192813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3242.283394] Lustre: lustre-MDT0000: new disk, initializing [ 3242.370223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3242.382187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3245.329515] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3253.575518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3253.636228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.713961] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3253.759952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3253.763370] Lustre: Skipped 1 previous similar message [ 3253.824785] Lustre: lustre-MDT0001: new disk, initializing [ 3253.921277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3253.961920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3253.967151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3257.418406] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3266.039605] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3270.675354] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3276.855356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3276.917481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3276.921578] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3277.094424] Lustre: lustre-OST0000: new disk, initializing [ 3277.099948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3277.157736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.734422] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3285.024270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3285.030225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3288.890349] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3291.931614] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3292.067335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3294.458926] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3294.600419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3295.713689] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3295.719356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3295.728512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3300.321815] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3300.326395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.335345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3301.793825] Lustre: server umount lustre-OST0000 complete [ 3305.443626] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3305.449847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.464064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3305.468300] Lustre: Skipped 1 previous similar message [ 3310.581194] Lustre: server umount lustre-MDT0000 complete [ 3311.074726] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3313.577898] LustreError: 154421:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316896 with bad export cookie 14644533569591933374 [ 3313.584032] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3313.588079] LustreError: 154421:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3313.813993] Lustre: server umount lustre-MDT0001 complete [ 3320.218296] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 10:41:41 (1761316901) [ 3333.198516] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3341.013647] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3349.058826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3354.335323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3358.989756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3363.720056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3368.507776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3368.569644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.721610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3368.760653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3368.815986] Lustre: lustre-MDT0000: new disk, initializing [ 3368.886217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.899195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3371.929760] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3379.318326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3379.364081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3379.417536] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3379.448738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3379.452350] Lustre: Skipped 1 previous similar message [ 3379.499788] Lustre: lustre-MDT0001: new disk, initializing [ 3379.565041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3379.586646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3379.592353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3382.721078] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3390.468124] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3395.546771] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3401.095249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3401.165279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.173986] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3401.347531] Lustre: lustre-OST0000: new disk, initializing [ 3401.357547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3401.438477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.729205] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3411.007887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3411.021552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3413.013196] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3417.669810] Lustre: Failing over lustre-OST0000 [ 3417.960543] Lustre: server umount lustre-OST0000 complete [ 3426.785944] Lustre: 152649:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317001/real 1761317001] req@0000000030a0beeb x1846874517568896/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1761317008 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3426.802308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.809403] Lustre: Skipped 1 previous similar message [ 3428.832507] Lustre: 152648:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317004/real 1761317004] req@000000006ac93c35 x1846874517569152/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761317011 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3433.953958] Lustre: 152647:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317009/real 1761317009] req@00000000e16bc728 x1846874517569984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761317016 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3434.013551] Lustre: 152647:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3437.410826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3437.823320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3437.844962] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3441.511900] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3443.558440] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3445.072425] Lustre: Failing over lustre-MDT0000 [ 3445.093353] LustreError: 152645:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3445.255901] Lustre: server umount lustre-MDT0000 complete [ 3445.988989] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.994564] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.015049] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 3446.016641] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 3446.020324] Lustre: Skipped 3 previous similar messages [ 3446.067517] LustreError: Skipped 3 previous similar messages [ 3448.692500] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.363720] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.374813] LustreError: Skipped 2 previous similar messages [ 3455.885913] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.897095] LustreError: Skipped 1 previous similar message [ 3458.465813] Lustre: 152647:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317033/real 1761317033] req@000000007acdf34f x1846874517576640/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761317040 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3458.485758] Lustre: 152647:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3458.493702] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3463.435958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.656471] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xcb3bd5b104cf7d0d to 0xcb3bd5b104cf81e4 [ 3463.671559] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 3463.798206] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3463.886380] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3467.106723] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3467.397489] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3469.296458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3469.319948] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3469.354839] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 3474.405802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.414831] Lustre: Skipped 1 previous similar message [ 3474.415784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3474.437735] Lustre: Skipped 2 previous similar messages [ 3476.332248] Lustre: server umount lustre-OST0000 complete [ 3479.522069] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3479.523151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3479.534497] Lustre: Skipped 1 previous similar message [ 3488.752147] LustreError: 161404:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317071 with bad export cookie 14644533569591935460 [ 3488.753561] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3488.763408] LustreError: 161404:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3499.068695] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3501.449893] Key type lgssc unregistered [ 3501.657628] LNet: 167258:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3501.672700] LNet: Removed LNI 192.168.203.123@tcp [ 3502.303331] Key type .llcrypt unregistered [ 3502.305105] Key type ._llcrypt unregistered [ 3520.918177] alg: No test for adler32 (adler32-zlib) [ 3521.669562] Key type ._llcrypt registered [ 3521.672681] Key type .llcrypt registered [ 3521.748921] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3522.350206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3522.610560] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 3522.746205] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3522.750981] LNet: Accept secure, port 988 [ 3524.426558] Key type lgssc registered [ 3525.226194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3530.856650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.198324] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3532.246227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.427633] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3537.378301] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3541.866884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3542.228991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3545.522898] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3550.856816] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3555.787564] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3561.107258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3561.376177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3562.413132] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 3564.254391] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3570.456740] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3588.992537] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3596.160599] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3627.418674] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3636.603368] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3650.453522] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3658.619125] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3681.653709] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3723.646913] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3723.654212] Lustre: Skipped 1 previous similar message [ 3799.468436] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3799.492537] Lustre: Skipped 5 previous similar messages [ 3929.480414] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3929.488087] Lustre: Skipped 2 previous similar messages [ 4224.399605] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 4224.411214] Lustre: Skipped 5 previous similar messages [ 4389.737374] Lustre: Failing over lustre-MDT0000 [ 4389.858766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.862986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.874806] Lustre: Skipped 2 previous similar messages [ 4389.891497] Lustre: Skipped 2 previous similar messages [ 4389.929398] LustreError: 171563:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000007ee01726) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4390.002605] Lustre: server umount lustre-MDT0000 complete [ 4390.758522] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.981421] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.990099] LustreError: Skipped 2 previous similar messages [ 4397.933958] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.952411] LustreError: Skipped 1 previous similar message [ 4400.099524] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.118144] LustreError: Skipped 2 previous similar messages [ 4402.080395] Lustre: 168030:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317977/real 1761317977] req@000000006dc239bc x1846874845123776/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761317984 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4402.095944] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4407.274678] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.290179] LustreError: Skipped 7 previous similar messages [ 4407.899838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.303602] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0x553624da0720cb83 to 0x553624da072337b2 [ 4408.316769] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 4408.510480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4408.568564] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4409.189478] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4411.641757] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4413.934515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4413.950637] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4413.976323] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 4418.606446] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4419.964987] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4439.522133] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4439.524354] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.533905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4439.541279] Lustre: Skipped 1 previous similar message [ 4441.585268] Lustre: server umount lustre-MDT0000 complete [ 4444.641609] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4444.656396] LustreError: Skipped 5 previous similar messages [ 4445.326084] LustreError: 169505:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318027 with bad export cookie 6140135660807468978 [ 4445.330267] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4445.331095] LustreError: 169505:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4449.764517] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.770648] Lustre: Skipped 1 previous similar message [ 4449.777313] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4449.779869] Lustre: Skipped 2 previous similar messages [ 4451.639567] Lustre: server umount lustre-MDT0001 complete [ 4460.557620] Lustre: server umount lustre-OST0000 complete [ 4467.125677] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4473.330090] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4480.775646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4485.350606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4489.653626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4493.844606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.725940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.777509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4498.953034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4498.968701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4499.021064] Lustre: lustre-MDT0000: new disk, initializing [ 4499.100790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4499.114077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4501.738775] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4508.007889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4508.056622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.111872] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4508.142373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4508.146059] Lustre: Skipped 1 previous similar message [ 4508.192333] Lustre: lustre-MDT0001: new disk, initializing [ 4508.269900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4508.301336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4508.308293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4510.949135] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4517.107095] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4520.877316] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4526.351821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4526.450186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4526.459939] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4526.785497] Lustre: lustre-OST0000: new disk, initializing [ 4526.788830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4526.859453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4530.694522] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4536.380555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4536.387857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4538.549652] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4541.936893] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4542.184362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4545.591176] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4545.727466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4552.162884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.163646] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4552.178941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4552.195074] Lustre: Skipped 2 previous similar messages [ 4553.758443] Lustre: server umount lustre-OST0000 complete [ 4557.282970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4557.287102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4557.296780] Lustre: Skipped 2 previous similar messages [ 4557.312891] Lustre: Skipped 1 previous similar message [ 4563.399684] Lustre: server umount lustre-MDT0000 complete [ 4566.597754] LustreError: 175692:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318149 with bad export cookie 6140135660807469699 [ 4566.605070] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4566.613905] LustreError: 175692:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4572.898439] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:02:34 (1761318154) [ 4584.333873] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4589.409818] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4594.837765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4599.034056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4603.734471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4608.793281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.891869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.940499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4613.088758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4613.108848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4613.168795] Lustre: lustre-MDT0000: new disk, initializing [ 4613.257796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4613.279446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4616.281710] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4624.566345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4624.621335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.680965] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4624.709583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4624.713425] Lustre: Skipped 1 previous similar message [ 4624.788687] Lustre: lustre-MDT0001: new disk, initializing [ 4624.856037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4624.885467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4624.889929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4628.048742] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4636.141895] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4641.064732] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4646.298615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4646.372964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4646.377356] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4646.575197] Lustre: lustre-OST0000: new disk, initializing [ 4646.588955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4646.647186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.612690] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4653.084874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4653.090682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4655.755442] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4678.464674] Lustre: DEBUG MARKER: conf-sanity test_49a: @@@@@@ FAIL: client mount failed [ 4685.097410] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:04:26 (1761318266) [ 4701.671701] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4701.683942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.699692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4701.705742] Lustre: Skipped 2 previous similar messages [ 4705.762831] LustreError: 185831:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000009bbef391) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 4705.789646] Lustre: server umount lustre-MDT0000 complete [ 4705.793462] Lustre: Skipped 1 previous similar message [ 4708.698739] LustreError: 184775:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318291 with bad export cookie 6140135660807470539 [ 4708.703192] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.712864] LustreError: 184775:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4709.348957] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4709.362209] LustreError: Skipped 2 previous similar messages [ 4724.951242] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4732.158738] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4739.843865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4745.353800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4749.779244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4754.507358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4759.190652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4759.246062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4759.416155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4759.438628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4759.484948] Lustre: lustre-MDT0000: new disk, initializing [ 4759.544825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4759.558778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4762.670442] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4771.085455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4771.134392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4771.205627] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4771.398578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4774.845403] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4783.238066] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4788.006228] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4793.098869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4793.152605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4793.157160] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4793.342965] Lustre: lustre-OST0000: new disk, initializing [ 4793.345093] Lustre: Skipped 1 previous similar message [ 4793.352341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4793.355291] Lustre: Skipped 2 previous similar messages [ 4796.914284] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4800.037819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4800.048115] Lustre: Skipped 1 previous similar message [ 4800.050862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4804.661905] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4810.728744] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4810.734314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4810.743379] LustreError: Skipped 1 previous similar message [ 4810.754318] Lustre: Skipped 3 previous similar messages [ 4810.763736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4810.770044] Lustre: Skipped 5 previous similar messages [ 4816.498150] Lustre: server umount lustre-OST0000 complete [ 4816.500323] Lustre: Skipped 2 previous similar messages [ 4828.773746] LustreError: 190782:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318411 with bad export cookie 6140135660807471519 [ 4828.778091] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4828.784254] LustreError: 190782:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4839.453363] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4841.816818] Key type lgssc unregistered [ 4842.097056] LNet: 192647:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4842.110055] LNet: Removed LNI 192.168.203.123@tcp [ 4842.812445] Key type .llcrypt unregistered [ 4842.814893] Key type ._llcrypt unregistered [ 4866.294722] alg: No test for adler32 (adler32-zlib) [ 4867.050419] Key type ._llcrypt registered [ 4867.052283] Key type .llcrypt registered [ 4867.158423] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4867.697532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4868.055317] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 4868.231018] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 4868.233927] LNet: Accept secure, port 988 [ 4869.848208] Key type lgssc registered [ 4870.546095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4876.867267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4878.417718] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4878.482493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.929569] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4883.939355] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4889.068036] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4889.721671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.199766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4893.900698] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4900.762979] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4906.600825] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4913.114810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4913.525086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4918.593364] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4926.360172] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4928.882145] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4949.474624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4949.477557] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4949.480072] Lustre: Skipped 1 previous similar message [ 4949.480772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4952.932786] Lustre: server umount lustre-OST0000 complete [ 4959.713628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4959.737192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4959.745791] Lustre: Skipped 2 previous similar messages [ 4963.186667] Lustre: server umount lustre-MDT0000 complete [ 4964.834182] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4964.852220] LustreError: Skipped 1 previous similar message [ 4967.157265] LustreError: 194102:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318549 with bad export cookie 5930705580969182627 [ 4967.169992] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.185713] LustreError: 194102:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4967.652558] Lustre: server umount lustre-MDT0001 complete [ 4978.714333] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4980.678156] Key type lgssc unregistered [ 4980.850812] LNet: 197798:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4980.863630] LNet: Removed LNI 192.168.203.123@tcp [ 4981.323341] Key type .llcrypt unregistered [ 4981.325056] Key type ._llcrypt unregistered [ 4999.481568] alg: No test for adler32 (adler32-zlib) [ 5000.237299] Key type ._llcrypt registered [ 5000.252690] Key type .llcrypt registered [ 5000.352977] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5001.231698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5001.655599] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 5001.820374] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5001.825691] LNet: Accept secure, port 988 [ 5003.456141] Key type lgssc registered [ 5004.468257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5011.007463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5012.351785] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5012.399036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5015.591127] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5017.572913] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5022.700362] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5023.504845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5023.960887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5029.198156] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5036.389777] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5041.734365] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5047.751927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5048.082669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5051.892170] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5058.147448] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5063.422038] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5068.773720] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5068.775979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5068.796407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5068.800764] Lustre: Skipped 1 previous similar message [ 5073.886423] Lustre: server umount lustre-OST0000 complete [ 5080.548515] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff989344899000.ost_server_uuid 40 [ 5081.992321] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5093.964846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5093.970282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5093.975157] Lustre: Skipped 2 previous similar messages [ 5093.978635] Lustre: Skipped 1 previous similar message [ 5098.976962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5098.985036] Lustre: Skipped 2 previous similar messages [ 5100.197197] Lustre: server umount lustre-MDT0000 complete [ 5101.025635] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.043582] LustreError: Skipped 1 previous similar message [ 5103.854964] LustreError: 199239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318686 with bad export cookie 17960654681947131169 [ 5103.864313] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5103.883735] LustreError: 199239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5104.275482] Lustre: server umount lustre-MDT0001 complete [ 5111.578031] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:11:32 (1761318692) [ 5117.142605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.529326] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5117.583183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5121.580428] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5128.169989] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.182170] LustreError: Skipped 1 previous similar message [ 5129.337431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5129.562806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5133.392375] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5139.914164] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5144.748767] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5150.556283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5150.812989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5153.929180] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5160.149613] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5165.811940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5165.871861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5165.876278] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5166.005950] Lustre: lustre-OST0001: new disk, initializing [ 5166.010697] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5168.199425] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 5169.697177] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5173.841671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5173.858320] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5177.689746] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5183.349113] Lustre: lustre-OST0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 5183.461588] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5183.474744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.330420] Lustre: server umount lustre-OST0000 complete [ 5188.453508] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.932162] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5191.139147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5194.569457] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5194.761953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5210.091152] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5210.092553] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5210.095278] LustreError: Skipped 1 previous similar message [ 5210.098447] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5210.121231] Lustre: Skipped 2 previous similar messages [ 5210.126303] Lustre: Skipped 4 previous similar messages [ 5213.079780] Lustre: server umount lustre-OST0001 complete [ 5216.738909] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5216.746816] LustreError: Skipped 1 previous similar message [ 5216.750034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.328730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5220.331852] Lustre: Skipped 3 previous similar messages [ 5222.228361] Lustre: server umount lustre-MDT0000 complete [ 5225.446396] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5225.457249] LustreError: Skipped 14 previous similar messages [ 5225.521854] LustreError: 203229:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318807 with bad export cookie 17960654681947132233 [ 5225.524571] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5225.535855] LustreError: 203229:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5230.331584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5234.049899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5237.537764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5240.900636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5246.443299] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:13:47 (1761318827) [ 5251.366314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.566429] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5251.592601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5251.726567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5251.734521] Lustre: Skipped 1 previous similar message [ 5255.151841] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5261.363491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5261.449235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5261.483978] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5264.626433] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5269.342419] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5273.353128] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5277.806460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5277.924456] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5281.356256] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5286.818542] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5287.000356] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 5291.680843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5291.735576] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5291.904552] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5291.907804] Lustre: Skipped 2 previous similar messages [ 5292.894965] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 5294.853496] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5300.375874] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5303.270247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.272584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5303.277070] Lustre: Skipped 2 previous similar messages [ 5303.288282] Lustre: Skipped 1 previous similar message [ 5308.443710] Lustre: server umount lustre-OST0000 complete [ 5308.448551] Lustre: Skipped 1 previous similar message [ 5311.331365] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.748316] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.748965] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5323.761619] Lustre: Skipped 1 previous similar message [ 5333.473510] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5333.476829] LustreError: Skipped 1 previous similar message [ 5339.550610] LustreError: 209663:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318921 with bad export cookie 17960654681947133619 [ 5339.563193] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5343.102708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5345.295138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5347.650613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5350.520643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5355.334362] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:15:37 (1761318937) [ 5366.276388] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5371.949432] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5377.964237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5381.313797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5384.425721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5387.920292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5391.530939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5391.578925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5391.690472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5391.708194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5391.755153] Lustre: lustre-MDT0000: new disk, initializing [ 5391.799904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5391.811238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5393.749147] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5399.776032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5399.818815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5399.867623] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5399.896844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5399.899235] Lustre: Skipped 1 previous similar message [ 5399.941809] Lustre: lustre-MDT0001: new disk, initializing [ 5400.002847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5400.007959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5402.393526] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5408.138059] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5411.168984] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5415.230899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5415.285032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5415.290603] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5415.461500] Lustre: lustre-OST0000: new disk, initializing [ 5415.465224] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5415.779596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5415.788598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5418.154613] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5425.436307] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5428.957155] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5429.100867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5432.354299] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5432.526392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5436.388621] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5436.389674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.401788] LustreError: Skipped 1 previous similar message [ 5436.406027] Lustre: Skipped 2 previous similar messages [ 5436.410874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5436.413565] Lustre: Skipped 7 previous similar messages [ 5448.160174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5448.344047] Lustre: server umount lustre-OST0000 complete [ 5448.347833] Lustre: Skipped 3 previous similar messages [ 5459.743962] LustreError: 217891:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319042 with bad export cookie 17960654681947135355 [ 5459.745098] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5459.754448] LustreError: 217891:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5466.093051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5466.512863] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5466.523226] LustreError: Skipped 9 previous similar messages [ 5469.395637] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5474.421540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5477.393926] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5482.478736] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5485.989501] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5490.028698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5492.848158] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5497.347917] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5500.018414] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5500.173401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5502.418676] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5506.693456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 5510.626337] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5510.628838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.633517] LustreError: Skipped 1 previous similar message [ 5510.646357] Lustre: Skipped 3 previous similar messages [ 5510.652583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5510.655193] Lustre: Skipped 8 previous similar messages [ 5513.914283] Lustre: server umount lustre-OST0000 complete [ 5513.916498] Lustre: Skipped 2 previous similar messages [ 5517.080282] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5519.436492] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5544.458665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5544.657380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5544.661940] Lustre: Skipped 5 previous similar messages [ 5546.975200] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5551.518230] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5552.354542] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5561.974087] LustreError: 223036:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319144 with bad export cookie 17960654681947136209 [ 5561.980758] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5561.986582] LustreError: 223036:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5566.703327] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:19:08 (1761319148) [ 5570.493055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.127958] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5577.604744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.071342] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5584.343931] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5587.409130] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5590.872989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5593.330923] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5597.555042] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5600.088443] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5602.281207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5604.621961] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5604.758739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5608.697662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5608.742197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5608.746781] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5608.834065] Lustre: lustre-OST0001: new disk, initializing [ 5608.836895] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5611.397422] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5614.113988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5614.124573] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5617.461150] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5620.407848] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 5620.541270] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5623.047141] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 5623.190033] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5624.290561] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5624.301878] LustreError: Skipped 1 previous similar message [ 5633.738402] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 5633.869535] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5636.527629] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 5636.687701] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5662.971338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5665.816883] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5668.326274] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5668.336325] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5668.337788] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 5670.720536] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5671.724273] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5673.446905] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.450366] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5673.455939] Lustre: Skipped 5 previous similar messages [ 5673.458166] Lustre: Skipped 7 previous similar messages [ 5674.959804] Lustre: server umount lustre-OST0001 complete [ 5674.962181] Lustre: Skipped 4 previous similar messages [ 5704.964891] LustreError: 228448:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319287 with bad export cookie 17960654681947137161 [ 5704.965815] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5704.974900] LustreError: 228448:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5714.690740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5717.404268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5719.611124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5722.385043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5726.372211] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:21:48 (1761319308) [ 5729.819562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5729.973764] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5729.992567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5732.155347] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5735.978353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5736.030409] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5737.991653] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5741.257923] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5743.868689] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5747.026426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5747.124295] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5749.274505] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5752.922870] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5763.349547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5763.395460] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5765.473956] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5769.113260] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5769.842694] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5771.845507] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 5771.951979] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5774.071470] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 5774.216308] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5777.024249] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9893492ba800.ost_server_uuid 40 [ 5777.698987] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9893492ba800.ost_server_uuid in FULL state after 0 sec [ 5778.121832] Lustre: Permanently deactivating lustre-OST0001 [ 5778.129375] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 5778.132673] Lustre: Skipped 1 previous similar message [ 5779.316711] Lustre: Permanently reactivating lustre-OST0001 [ 5784.035823] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5784.040756] LustreError: Skipped 4 previous similar messages [ 5789.154462] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5789.160535] LustreError: Skipped 32 previous similar messages [ 5799.833562] LustreError: 238181:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319382 with bad export cookie 17960654681947138435 [ 5799.837292] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5799.838820] LustreError: 238181:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5802.331556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5804.073139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5805.649424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5807.295385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5810.167371] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:23:12 (1761319392) [ 5810.563289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5813.240100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.379995] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5813.394553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5813.398404] Lustre: Skipped 5 previous similar messages [ 5813.477794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5813.480421] Lustre: Skipped 9 previous similar messages [ 5815.171442] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5818.820852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5818.886448] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5820.842904] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5823.913312] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5826.595668] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5829.760613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5829.791469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5829.898862] Lustre: Permanently deactivating lustre-OST0000 [ 5829.938050] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5831.862491] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5835.322824] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5838.097241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5839.022359] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 5839.990740] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5842.910870] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5843.545224] Lustre: Permanently reactivating lustre-OST0000 [ 5848.592859] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 5874.700526] LustreError: 244217:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319457 with bad export cookie 17960654681947140017 [ 5874.704234] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5874.707842] LustreError: 244217:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5880.889176] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5882.086287] Key type lgssc unregistered [ 5882.218623] LNet: 249183:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5882.222579] LNet: Removed LNI 192.168.203.123@tcp [ 5882.549799] Key type .llcrypt unregistered [ 5882.551551] Key type ._llcrypt unregistered [ 5892.159606] alg: No test for adler32 (adler32-zlib) [ 5892.911376] Key type ._llcrypt registered [ 5892.913056] Key type .llcrypt registered [ 5892.961330] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5893.376603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5893.615695] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 5893.707694] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5893.710592] LNet: Accept secure, port 988 [ 5895.304205] Key type lgssc registered [ 5895.784755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5897.183045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5899.974216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.197754] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5901.230588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5903.114211] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5906.401518] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5906.696687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5906.730441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.795889] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5906.811441] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5906.815528] Lustre: Permanently deactivating lustre-MDT0001 [ 5906.847385] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5906.850949] LustreError: 250726:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5906.856745] LustreError: 250726:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5906.861887] LustreError: 250726:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 5906.955441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5908.786582] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5911.856529] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5914.100029] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5916.774742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5916.918312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5918.630610] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5921.635745] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5922.020735] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5922.023482] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:131 to 0x280000400:161 [ 5924.447332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5924.552370] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5926.005543] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 [ 5926.340236] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 5929.356671] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5931.114733] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 5931.115154] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:129 [ 5935.500763] Lustre: Permanently reactivating lustre-MDT0001 [ 5935.509530] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 5935.512964] Lustre: Skipped 3 previous similar messages [ 5938.727076] Lustre: lustre-MDT0001: Client 57925c44-1545-45ec-b4b2-a94e25d0f67d (at 192.168.203.23@tcp) reconnecting [ 5941.943358] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5942.576493] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5944.831345] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 5944.916996] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5946.700298] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 5946.791958] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5947.893993] Lustre: Permanently deactivating lustre-MDT0001 [ 5947.901528] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 5947.904634] Lustre: Skipped 2 previous similar messages [ 5953.025432] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5967.329979] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.330899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5967.335148] Lustre: Skipped 2 previous similar messages [ 5967.340593] Lustre: Skipped 3 previous similar messages [ 5970.388204] Lustre: server umount lustre-MDT0000 complete [ 5971.881153] LustreError: 250676:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319554 with bad export cookie 11939358478296511335 [ 5971.883261] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5971.885769] LustreError: 250676:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5972.032724] Lustre: server umount lustre-MDT0001 complete [ 5979.508534] Lustre: server umount lustre-OST0000 complete [ 5983.510429] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:26:05 (1761319565) [ 5987.280682] Lustre: server umount lustre-OST0001 complete [ 5989.637478] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5992.953547] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5997.169413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6000.002822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6002.449393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6005.450072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6009.906881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6009.962796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6010.094411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6010.096770] Lustre: Skipped 2 previous similar messages [ 6010.118318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6010.171149] Lustre: lustre-MDT0000: new disk, initializing [ 6010.304179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6010.323560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6015.787790] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6028.486594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6028.584077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6028.713759] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6028.867498] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6028.879832] Lustre: Skipped 1 previous similar message [ 6029.002208] Lustre: lustre-MDT0001: new disk, initializing [ 6029.129189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6029.250603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6029.266837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6034.398868] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6045.866974] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6053.118897] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6060.855867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6060.987459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6061.005958] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6061.319708] Lustre: lustre-OST0000: new disk, initializing [ 6061.322948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6061.411169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6066.335220] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6070.335863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6070.352908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6075.639242] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6077.479575] LustreError: 259196:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6085.581515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6085.627122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6085.631745] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6085.714995] Lustre: lustre-OST0001: new disk, initializing [ 6085.719484] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6087.536072] LustreError: 259196:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 6087.562755] LustreError: 259196:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6089.430094] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6091.828443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6091.833688] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6097.122292] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6097.608217] LustreError: 259196:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 6098.741979] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6101.986980] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6101.988395] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.996508] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6102.009340] Lustre: Skipped 2 previous similar messages [ 6102.013871] Lustre: Skipped 1 previous similar message [ 6106.452461] Lustre: server umount lustre-OST0001 complete [ 6107.114890] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6112.230018] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6112.235481] LustreError: Skipped 3 previous similar messages [ 6121.953145] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6121.956730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.969403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6121.976689] Lustre: Skipped 2 previous similar messages [ 6122.470679] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.479939] LustreError: Skipped 3 previous similar messages [ 6127.585803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6127.598110] Lustre: Skipped 2 previous similar messages [ 6132.707716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6132.707716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6135.264472] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6135.446198] Lustre: server umount lustre-OST0000 complete [ 6141.921437] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6141.926343] LustreError: Skipped 1 previous similar message [ 6141.928951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.936178] Lustre: Skipped 1 previous similar message [ 6141.942170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6147.614080] LustreError: 259180:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319730 with bad export cookie 11939358478296514107 [ 6147.617907] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6147.619553] LustreError: 259180:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6153.952560] Lustre: server umount lustre-MDT0001 complete [ 6153.955809] Lustre: Skipped 1 previous similar message [ 6162.722963] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6164.677819] Key type lgssc unregistered [ 6164.836544] LNet: 264392:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6164.845668] LNet: Removed LNI 192.168.203.123@tcp [ 6165.332454] Key type .llcrypt unregistered [ 6165.334930] Key type ._llcrypt unregistered [ 6189.153651] alg: No test for adler32 (adler32-zlib) [ 6189.907587] Key type ._llcrypt registered [ 6189.910183] Key type .llcrypt registered [ 6190.011698] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 6203.711529] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6204.545821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6204.868760] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 6205.078861] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6205.082721] LNet: Accept secure, port 988 [ 6206.736329] Key type lgssc registered [ 6207.626653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6214.931628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6219.785398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6224.693575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6230.105314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6235.737238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6235.806142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6236.972144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6237.005181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6237.114278] Lustre: lustre-MDT0000: new disk, initializing [ 6237.205875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6237.241840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6240.430821] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6248.873267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6248.949614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.030821] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6249.076601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6249.079610] Lustre: Skipped 1 previous similar message [ 6249.154915] Lustre: lustre-MDT0001: new disk, initializing [ 6249.262458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6249.290615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6249.294762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6252.277980] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6260.431680] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6265.484727] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6271.682246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6271.755283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.761559] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6271.936996] Lustre: lustre-OST0000: new disk, initializing [ 6271.942327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6272.009786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6275.428886] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6278.166391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6278.171937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6282.143342] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6285.557283] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6285.720991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6289.265620] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6289.425239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6293.474557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6293.485716] Lustre: Skipped 1 previous similar message [ 6293.490908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6298.602372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6298.610234] Lustre: Skipped 1 previous similar message [ 6303.715475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6303.726022] Lustre: Skipped 1 previous similar message [ 6305.248139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6305.435162] Lustre: server umount lustre-OST0000 complete [ 6308.835481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6308.840383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6308.859398] Lustre: Skipped 1 previous similar message [ 6308.869060] Lustre: Skipped 2 previous similar messages [ 6313.953355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6313.976731] Lustre: Skipped 1 previous similar message [ 6314.668485] Lustre: server umount lustre-MDT0000 complete [ 6318.100805] LustreError: 271978:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319900 with bad export cookie 17023324817465054428 [ 6318.106136] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6318.125354] LustreError: 271978:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6318.403929] Lustre: server umount lustre-MDT0001 complete [ 6326.925971] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 11:31:48 (1761319908) [ 6332.804275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.138069] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6333.216591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6336.728517] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6338.534347] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6343.600483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6343.650063] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6343.954745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6347.240724] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6352.542453] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6356.811434] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6361.221166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6361.570170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6364.480533] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6369.696269] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6388.195658] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6388.204959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6388.215159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6389.220215] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6390.567851] Lustre: server umount lustre-OST0000 complete [ 6394.271122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6404.570656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6404.853756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6408.681159] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6409.959255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.967846] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 6409.976587] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 6414.423677] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6421.234319] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.254730] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 6421.267387] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 6425.575271] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6425.579164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.584988] Lustre: Skipped 1 previous similar message [ 6425.589154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.592106] Lustre: Skipped 2 previous similar messages [ 6428.401404] Lustre: server umount lustre-OST0000 complete [ 6431.716219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6431.724580] Lustre: Skipped 1 previous similar message [ 6436.321952] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6436.339784] LustreError: Skipped 1 previous similar message [ 6437.744976] Lustre: server umount lustre-MDT0000 complete [ 6441.200435] LustreError: 273035:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320023 with bad export cookie 17023324817465055275 [ 6441.205789] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6441.217358] LustreError: 273035:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6451.959979] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6454.426186] Key type lgssc unregistered [ 6454.723518] LNet: 278482:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6454.734360] LNet: Removed LNI 192.168.203.123@tcp [ 6455.311504] Key type .llcrypt unregistered [ 6455.312954] Key type ._llcrypt unregistered [ 6472.692056] alg: No test for adler32 (adler32-zlib) [ 6473.445496] Key type ._llcrypt registered [ 6473.447758] Key type .llcrypt registered [ 6473.574846] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6474.246639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6474.758741] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 6474.953595] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6474.958613] LNet: Accept secure, port 988 [ 6476.608121] Key type lgssc registered [ 6477.457150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6483.940231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.307529] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6485.368935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6489.024308] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6490.605608] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6495.714737] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6496.540479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.997457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6500.783901] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6506.998472] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6511.532790] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6516.710192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6516.974576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6520.455758] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6527.017228] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6536.430674] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 6553.059494] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6553.063908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.077647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6557.153990] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6557.156304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6557.158261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6557.172872] Lustre: Skipped 1 previous similar message [ 6558.449888] Lustre: server umount lustre-OST0000 complete [ 6562.275630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6562.280966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6567.221756] Lustre: server umount lustre-MDT0000 complete [ 6567.393772] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.409376] LustreError: Skipped 1 previous similar message [ 6570.338603] LustreError: 279928:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320152 with bad export cookie 18289412010218411032 [ 6570.343706] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6570.345323] LustreError: 279928:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6570.595815] Lustre: server umount lustre-MDT0001 complete [ 6579.391342] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6581.490824] Key type lgssc unregistered [ 6581.737058] LNet: 284394:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6581.744873] LNet: Removed LNI 192.168.203.123@tcp [ 6582.451765] Key type .llcrypt unregistered [ 6582.454259] Key type ._llcrypt unregistered [ 6593.605487] alg: No test for adler32 (adler32-zlib) [ 6594.358354] Key type ._llcrypt registered [ 6594.360647] Key type .llcrypt registered [ 6594.439301] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6595.071406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6595.433601] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 6595.631539] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6595.635031] LNet: Accept secure, port 988 [ 6597.272171] Key type lgssc registered [ 6598.195709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6604.134523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.532461] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6605.592184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6609.181975] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6609.574961] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6615.011714] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6615.300459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6615.540694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6618.590806] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6624.265539] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6629.315957] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6634.556960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6634.827653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6635.880664] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 6637.502053] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6641.917697] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6654.949933] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6654.957135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6654.968465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6655.465522] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6655.471598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6655.479762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6655.491292] Lustre: Skipped 1 previous similar message [ 6659.702377] Lustre: server umount lustre-OST0000 complete [ 6665.697415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6665.700926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6665.712794] Lustre: Skipped 1 previous similar message [ 6668.732644] Lustre: server umount lustre-MDT0000 complete [ 6670.817547] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6670.823859] LustreError: Skipped 1 previous similar message [ 6671.678245] LustreError: 285642:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320254 with bad export cookie 1294578321694301139 [ 6671.684348] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6671.687311] LustreError: 285642:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6672.035701] Lustre: server umount lustre-MDT0001 complete [ 6681.385804] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6683.175896] Key type lgssc unregistered [ 6683.402735] LNet: 289484:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6683.416991] LNet: Removed LNI 192.168.203.123@tcp [ 6684.024763] Key type .llcrypt unregistered [ 6684.026376] Key type ._llcrypt unregistered [ 6695.397564] alg: No test for adler32 (adler32-zlib) [ 6696.151369] Key type ._llcrypt registered [ 6696.153342] Key type .llcrypt registered [ 6696.221190] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6696.702579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6696.964737] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 6697.112136] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6697.116113] LNet: Accept secure, port 988 [ 6698.744124] Key type lgssc registered [ 6699.400259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6705.473412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.855439] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6706.917133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6710.293378] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6712.290045] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6716.473420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6716.683753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6719.692328] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6726.567679] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6731.187454] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6736.490859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6736.843717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6737.848348] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 6740.043951] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6745.070344] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6757.351434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6757.355570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6757.373564] Lustre: Skipped 1 previous similar message [ 6757.381038] Lustre: Skipped 1 previous similar message [ 6760.474661] Lustre: server umount lustre-OST0000 complete [ 6767.586339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6767.589245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6767.599309] Lustre: Skipped 1 previous similar message [ 6767.613717] Lustre: Skipped 1 previous similar message [ 6769.876609] Lustre: server umount lustre-MDT0000 complete [ 6772.676725] LustreError: 290731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320355 with bad export cookie 7318367123530359662 [ 6772.677390] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6772.690159] LustreError: 290731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6773.032821] Lustre: server umount lustre-MDT0001 complete [ 6782.129281] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6784.154904] Key type lgssc unregistered [ 6784.399641] LNet: 294401:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6784.409059] LNet: Removed LNI 192.168.203.123@tcp [ 6784.927192] Key type .llcrypt unregistered [ 6784.929639] Key type ._llcrypt unregistered [ 6801.844435] alg: No test for adler32 (adler32-zlib) [ 6802.601573] Key type ._llcrypt registered [ 6802.605225] Key type .llcrypt registered [ 6802.701142] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6803.286911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6803.651268] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 6803.807573] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6803.811987] LNet: Accept secure, port 988 [ 6805.448143] Key type lgssc registered [ 6806.425470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6813.012042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6814.313298] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6814.364246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6817.665549] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6819.811924] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.395588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6823.590414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6826.429236] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6831.937112] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6836.373137] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6841.058914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6841.250685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6844.225282] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6845.363438] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 6849.833068] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6871.012058] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6871.015762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6871.025415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6875.107693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6875.108026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6876.294399] Lustre: server umount lustre-OST0000 complete [ 6880.225716] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6880.228292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6880.231881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6880.231889] Lustre: Skipped 1 previous similar message [ 6880.253979] Lustre: Skipped 1 previous similar message [ 6885.346659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6885.349571] Lustre: Skipped 1 previous similar message [ 6885.489240] Lustre: server umount lustre-MDT0000 complete [ 6887.926345] LustreError: 295858:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320470 with bad export cookie 11350877984821101448 [ 6887.927252] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6887.931872] LustreError: 295858:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6888.213936] Lustre: server umount lustre-MDT0001 complete [ 6896.264122] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6898.191843] Key type lgssc unregistered [ 6898.409296] LNet: 300352:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6898.423044] LNet: Removed LNI 192.168.203.123@tcp [ 6898.919564] Key type .llcrypt unregistered [ 6898.921279] Key type ._llcrypt unregistered [ 6909.179919] alg: No test for adler32 (adler32-zlib) [ 6909.931566] Key type ._llcrypt registered [ 6909.934175] Key type .llcrypt registered [ 6910.020360] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6910.692385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6910.977289] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 6911.105814] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6911.109386] LNet: Accept secure, port 988 [ 6912.720604] Key type lgssc registered [ 6913.434146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6919.016433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6920.421459] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6920.460830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6923.499546] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6925.794515] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.961648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.196444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6933.095591] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6938.822510] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6943.100760] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6947.247406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6947.474682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6949.956234] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 6951.601779] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 6954.993468] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6966.762182] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6966.766077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6966.784955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6970.337927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6970.339310] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6970.339444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.345076] Lustre: Skipped 1 previous similar message [ 6972.240470] Lustre: server umount lustre-OST0000 complete [ 6975.458594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6975.459396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6975.464328] Lustre: Skipped 1 previous similar message [ 6980.511727] Lustre: server umount lustre-MDT0000 complete [ 6980.578045] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6980.589380] LustreError: Skipped 1 previous similar message [ 6982.557036] LustreError: 301598:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320564 with bad export cookie 4962123680658475784 [ 6982.559733] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6982.565629] LustreError: 301598:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6982.741808] Lustre: server umount lustre-MDT0001 complete [ 6988.678481] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6990.113036] Key type lgssc unregistered [ 6990.254472] LNet: 305452:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6990.260588] LNet: Removed LNI 192.168.203.123@tcp [ 6990.701650] Key type .llcrypt unregistered [ 6990.703455] Key type ._llcrypt unregistered [ 7000.252177] alg: No test for adler32 (adler32-zlib) [ 7001.004318] Key type ._llcrypt registered [ 7001.006286] Key type .llcrypt registered [ 7001.061133] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7001.560634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7001.780272] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 7001.899400] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7001.901817] LNet: Accept secure, port 988 [ 7003.521313] Key type lgssc registered [ 7004.122558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7008.313785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7009.550716] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7009.577498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7011.688197] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7013.602973] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.751398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7015.958327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7018.250935] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7021.969718] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7024.994941] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7028.103829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7028.315790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7030.655639] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7034.396167] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7039.596716] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 7045.091677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7045.091719] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7045.098622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7045.103583] Lustre: Skipped 1 previous similar message [ 7049.125558] Lustre: server umount lustre-OST0000 complete [ 7051.747784] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7051.751977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7051.759079] Lustre: Skipped 1 previous similar message [ 7051.763593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7055.331379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7055.333058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7057.389342] Lustre: server umount lustre-MDT0000 complete [ 7059.815120] LustreError: 306699:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320642 with bad export cookie 9024508486521291539 [ 7059.818764] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7059.826546] LustreError: 306699:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7060.090623] Lustre: server umount lustre-MDT0001 complete [ 7067.076875] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7068.583318] Key type lgssc unregistered [ 7068.736362] LNet: 310372:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7068.740799] LNet: Removed LNI 192.168.203.123@tcp [ 7069.181371] Key type .llcrypt unregistered [ 7069.183199] Key type ._llcrypt unregistered [ 7076.123533] alg: No test for adler32 (adler32-zlib) [ 7076.879438] Key type ._llcrypt registered [ 7076.882618] Key type .llcrypt registered [ 7076.938380] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7087.620180] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7095.408414] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7095.763843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7096.039767] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 7096.143416] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7096.146154] LNet: Accept secure, port 988 [ 7097.745758] Key type lgssc registered [ 7098.307700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7103.973419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7108.009910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7111.703762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7115.783443] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7120.112232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7120.151579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.325671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7121.347861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7121.409732] Lustre: lustre-MDT0000: new disk, initializing [ 7121.479377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7121.496817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7124.603792] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7131.822392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7131.871140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7131.921731] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7131.964130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7131.968401] Lustre: Skipped 1 previous similar message [ 7132.032473] Lustre: lustre-MDT0001: new disk, initializing [ 7132.123110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7132.147938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7132.153300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7134.923183] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7141.963968] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7145.623209] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7149.539394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7149.578793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7149.585139] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7149.709786] Lustre: lustre-OST0000: new disk, initializing [ 7149.712908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7149.773994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7152.202690] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7158.305997] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7159.826974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7159.831290] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7160.757237] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7160.873657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7163.363796] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7163.525208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7164.897724] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7164.901656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7164.908443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7165.409320] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7165.413067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7165.413168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7166.855784] Lustre: server umount lustre-OST0000 complete [ 7170.529484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7170.532640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7170.537964] Lustre: Skipped 1 previous similar message [ 7170.541014] Lustre: Skipped 1 previous similar message [ 7175.650880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7175.655824] Lustre: Skipped 1 previous similar message [ 7180.770393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7180.773296] Lustre: Skipped 1 previous similar message [ 7183.328116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7183.414157] Lustre: server umount lustre-MDT0000 complete [ 7185.551284] LustreError: 314113:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320767 with bad export cookie 15830781694815507945 [ 7185.553834] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7185.556168] LustreError: 314113:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7185.732990] Lustre: server umount lustre-MDT0001 complete [ 7190.349851] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:46:12 (1761320772) [ 7193.516864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7193.731518] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7193.771161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7195.694311] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7199.202416] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7199.988587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.158656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7202.139794] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7205.870459] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7208.426357] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7211.555712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7211.744712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7213.952697] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7218.018612] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7234.530335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.533393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7234.533843] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7234.536070] Lustre: Skipped 1 previous similar message [ 7234.549469] Lustre: Skipped 1 previous similar message [ 7240.011502] Lustre: server umount lustre-OST0000 complete [ 7242.284370] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7242.289744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7248.495224] Lustre: server umount lustre-MDT0000 complete [ 7249.894712] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7249.907271] LustreError: Skipped 1 previous similar message [ 7250.889880] LustreError: 319454:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320833 with bad export cookie 15830781694815508799 [ 7250.891318] LustreError: 319457:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 7250.891980] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7250.899207] LustreError: 319454:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7250.903108] LustreError: 319457:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 7250.922705] LustreError: 319457:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 7258.534449] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7260.189831] Key type lgssc unregistered [ 7260.373747] LNet: 322347:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7260.380940] LNet: Removed LNI 192.168.203.123@tcp [ 7260.823691] Key type .llcrypt unregistered [ 7260.825402] Key type ._llcrypt unregistered [ 7277.185592] alg: No test for adler32 (adler32-zlib) [ 7277.939962] Key type ._llcrypt registered [ 7277.944393] Key type .llcrypt registered [ 7278.016431] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7286.951852] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7287.568588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7287.837761] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 7287.971730] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7287.976195] LNet: Accept secure, port 988 [ 7289.608115] Key type lgssc registered [ 7290.240556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7295.401352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7299.369264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7303.113050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7306.982198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7309.696740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7312.327072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7315.765306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7315.797767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.926109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7316.945660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7316.992789] Lustre: lustre-MDT0000: new disk, initializing [ 7317.036540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7317.048075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7319.007312] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7324.953339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7324.986769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.034416] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7325.071546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7325.074143] Lustre: Skipped 1 previous similar message [ 7325.113386] Lustre: lustre-MDT0001: new disk, initializing [ 7325.167102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7325.190502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7325.195739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7327.273434] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7332.690436] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7335.639409] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7339.135341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7339.188291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7339.195322] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7339.383563] Lustre: lustre-OST2710: new disk, initializing [ 7339.388595] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 7339.448583] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7341.451666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7341.456870] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7341.665707] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7346.904772] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7350.388767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7350.433078] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7350.439091] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7350.517204] Lustre: lustre-OST03e8: new disk, initializing [ 7350.522106] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 7350.578569] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7352.841393] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7357.109602] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7359.531059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7359.536686] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7361.146286] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 7361.276937] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7363.511491] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 7363.630404] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7365.876146] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 7366.004714] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7368.552456] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 7368.672434] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7371.233178] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7371.237418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7371.247165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7374.820025] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7374.821275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7374.827930] Lustre: Skipped 2 previous similar messages [ 7374.830798] Lustre: Skipped 2 previous similar messages [ 7376.673268] Lustre: server umount lustre-MDT0000 complete [ 7378.525022] LustreError: 327140:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320960 with bad export cookie 14740008734028739877 [ 7378.525790] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7378.530970] LustreError: 327140:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7378.688444] Lustre: server umount lustre-MDT0001 complete [ 7387.104170] Lustre: 331770:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320963/real 1761320963] req@0000000077eee5c9 x1846878792606784/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761320969 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7387.104235] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7387.114258] Lustre: 331770:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7387.154259] Lustre: server umount lustre-OST2710 complete [ 7398.412294] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7403.306548] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7408.113985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7411.319870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7414.281220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7417.329040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7420.866607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7420.902794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7421.018523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7421.034297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7421.079319] Lustre: lustre-MDT0000: new disk, initializing [ 7421.118924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7421.128994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7423.180039] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7429.236723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7429.271050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7429.321404] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7429.353491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7429.356912] Lustre: Skipped 1 previous similar message [ 7429.410317] Lustre: lustre-MDT0001: new disk, initializing [ 7429.473482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7429.503682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7429.508366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7431.509490] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7436.400981] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7438.968099] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7442.393203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7442.433164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7442.436655] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7444.826976] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7449.609252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7449.615519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7449.689751] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7451.685700] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7451.787292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7453.926354] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7454.080594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7459.811770] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7459.815216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.815951] LustreError: Skipped 1 previous similar message [ 7459.823290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7459.827707] Lustre: Skipped 1 previous similar message [ 7461.122755] Lustre: server umount lustre-OST0000 complete [ 7461.124941] Lustre: Skipped 1 previous similar message [ 7464.929534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7464.930215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7464.938650] Lustre: Skipped 2 previous similar messages [ 7464.943555] Lustre: Skipped 2 previous similar messages [ 7469.137733] Lustre: server umount lustre-MDT0000 complete [ 7470.051626] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7470.059983] LustreError: Skipped 1 previous similar message [ 7470.871917] LustreError: 336959:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321053 with bad export cookie 14740008734028741487 [ 7470.872288] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7470.876714] LustreError: 336959:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7475.567638] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:50:57 (1761321057) [ 7476.353169] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7477.256410] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:50:59 (1761321059) [ 7479.063843] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7482.674214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7484.403435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7486.143017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7487.768989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7488.256450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7491.261178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7491.402307] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7491.417193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7491.519294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7491.524373] Lustre: Skipped 1 previous similar message [ 7493.694461] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7497.625427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.695304] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7499.703435] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7503.200944] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7505.906407] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7508.829130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7508.858831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7508.949639] Lustre: Denying initial registration attempt from nid 192.168.203.123@tcp, specified as failover [ 7508.955855] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7508.961516] LustreError: 342887:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 7508.964923] LustreError: 342887:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 7508.968200] LustreError: 342887:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 7509.023604] Lustre: server umount lustre-OST0000 complete [ 7509.025847] Lustre: Skipped 1 previous similar message [ 7509.027823] LustreError: 342887:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7513.057511] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7513.061267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.067784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7519.092496] LustreError: 340928:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321101 with bad export cookie 14740008734028742341 [ 7519.094507] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7519.099613] LustreError: 340928:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7524.959996] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7526.254306] Key type lgssc unregistered [ 7526.400549] LNet: 343962:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7526.406477] LNet: Removed LNI 192.168.203.123@tcp [ 7526.767663] Key type .llcrypt unregistered [ 7526.769412] Key type ._llcrypt unregistered [ 7532.266833] alg: No test for adler32 (adler32-zlib) [ 7533.018350] Key type ._llcrypt registered [ 7533.019716] Key type .llcrypt registered [ 7533.061835] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7533.447691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7533.685430] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 7533.788898] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7533.791791] LNet: Accept secure, port 988 [ 7535.392318] Key type lgssc registered [ 7535.885080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7539.161783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7540.859956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7542.635448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7544.440474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7544.998378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7552.577241] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7556.463610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7557.601332] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7557.616855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7557.692483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7559.711924] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7563.648756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7563.708385] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7563.722558] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7563.819961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7565.577705] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7568.949409] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7571.696686] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7574.846107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7574.878273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7574.969173] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7575.050716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7576.964410] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7580.722603] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7586.274564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7586.276411] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7586.277285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7586.280281] Lustre: Skipped 1 previous similar message [ 7586.283891] LustreError: Skipped 1 previous similar message [ 7591.397483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7591.400669] Lustre: Skipped 2 previous similar messages [ 7596.000767] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7596.085608] Lustre: server umount lustre-OST0000 complete [ 7599.585430] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7599.588975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7599.595311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7601.634071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7601.634685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7603.789466] Lustre: server umount lustre-MDT0000 complete [ 7605.476186] LustreError: 347633:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321187 with bad export cookie 9281736862332657111 [ 7605.478377] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7605.481689] LustreError: 347633:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7605.635891] Lustre: server umount lustre-MDT0001 complete [ 7610.377671] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7611.488246] Key type lgssc unregistered [ 7611.625584] LNet: 350532:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7611.629958] LNet: Removed LNI 192.168.203.123@tcp [ 7611.965854] Key type .llcrypt unregistered [ 7611.968241] Key type ._llcrypt unregistered [ 7622.976247] alg: No test for adler32 (adler32-zlib) [ 7623.728376] Key type ._llcrypt registered [ 7623.730020] Key type .llcrypt registered [ 7623.773981] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7624.205163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7624.375722] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 7624.472564] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7624.474509] LNet: Accept secure, port 988 [ 7626.072129] Key type lgssc registered [ 7626.513089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7629.741369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7630.947787] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7630.974768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7632.713393] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7636.161537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7636.324249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7638.122175] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7641.529779] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7643.969337] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7646.752256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7646.898945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7648.621800] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7652.052345] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7665.188243] LustreError: 354658:0:(ldlm_resource.c:1127:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200001b71:0x1:0x0].0xf14dc48e (00000000a7bf79f5) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7666.657435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7666.659607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7666.662434] Lustre: Skipped 2 previous similar messages [ 7666.665230] Lustre: Skipped 1 previous similar message [ 7667.562633] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 7667.566189] Lustre: Skipped 1 previous similar message [ 7671.304624] Lustre: server umount lustre-MDT0000 complete [ 7671.778772] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7671.785527] LustreError: Skipped 2 previous similar messages [ 7672.871250] LustreError: 354660:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321255 with bad export cookie 9465004486684721026 [ 7672.876519] LustreError: 354660:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7672.877208] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7676.897064] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7676.897262] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7676.902726] LustreError: Skipped 1 previous similar message [ 7676.908980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7678.946346] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7678.946603] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7678.951772] LustreError: Skipped 2 previous similar messages [ 7678.957675] Lustre: Skipped 2 previous similar messages [ 7684.065114] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7684.065172] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7684.068086] Lustre: Skipped 1 previous similar message [ 7684.076074] LustreError: Skipped 1 previous similar message [ 7687.136128] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7687.255955] Lustre: server umount lustre-MDT0001 complete [ 7689.945620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7693.527340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7706.529840] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0x835a736ed2523f82 to 0x835a736ed2525b74 [ 7706.535594] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 7706.647806] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7706.655826] LustreError: Skipped 2 previous similar messages [ 7706.678693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7706.703294] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 7708.415752] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7712.035854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7712.186715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7714.000849] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7714.214262] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 7717.352421] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7718.101741] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7720.771487] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7721.578934] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7724.513903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7724.518554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7724.520620] Lustre: Skipped 1 previous similar message [ 7724.522581] Lustre: Skipped 2 previous similar messages [ 7729.835337] Lustre: server umount lustre-OST0000 complete [ 7732.706889] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7732.710426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7737.605291] Lustre: server umount lustre-MDT0000 complete [ 7739.216730] LustreError: 354660:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321321 with bad export cookie 9465004486684728180 [ 7739.217176] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7739.221358] LustreError: 354660:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7745.019756] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7746.131544] Key type lgssc unregistered [ 7746.264557] LNet: 358502:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7746.268416] LNet: Removed LNI 192.168.203.123@tcp [ 7746.606559] Key type .llcrypt unregistered [ 7746.608295] Key type ._llcrypt unregistered [ 7757.522262] alg: No test for adler32 (adler32-zlib) [ 7758.274457] Key type ._llcrypt registered [ 7758.275836] Key type .llcrypt registered [ 7758.321190] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7758.724944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7758.930164] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 7759.018468] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7759.020990] LNet: Accept secure, port 988 [ 7760.616129] Key type lgssc registered [ 7761.089466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7764.673985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7765.885318] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7765.909696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.674989] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7771.105647] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7771.672453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7771.838327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7773.821800] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7777.751793] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7780.835483] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7786.466349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7786.478118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7787.495892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7787.498581] Lustre: Skipped 2 previous similar messages [ 7788.627071] Lustre: server umount lustre-MDT0000 complete [ 7790.710386] LustreError: 361689:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321373 with bad export cookie 13473624938628892994 [ 7790.713202] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7790.716189] LustreError: 361689:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7790.909184] Lustre: server umount lustre-MDT0001 complete [ 7797.937108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7798.075987] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7798.090855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7798.185271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.144183] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7804.030149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.085214] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7804.100807] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7804.217605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7806.328475] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7810.156304] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7812.982281] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7816.552926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7816.644728] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7816.649413] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7816.654818] LustreError: 364650:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 7816.658415] LustreError: 364650:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 7816.661758] LustreError: 364650:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 7816.720765] Lustre: server umount lustre-OST0000 complete [ 7816.722767] LustreError: 364650:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7819.713569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7819.815760] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7819.888478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.721495] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7824.958807] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 7825.164462] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7829.100285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7829.148669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7829.152457] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7829.195625] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7829.245574] Lustre: lustre-OST0001: new disk, initializing [ 7829.248789] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7829.303081] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7831.766344] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7832.663442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7832.668081] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7836.056906] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7838.178132] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7838.178259] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7838.182962] LustreError: Skipped 1 previous similar message [ 7838.193377] Lustre: Skipped 1 previous similar message [ 7838.196566] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7838.211547] Lustre: Skipped 1 previous similar message [ 7839.706752] Lustre: server umount lustre-OST0001 complete [ 7843.298793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7843.299053] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7843.309917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7843.310860] LustreError: Skipped 1 previous similar message [ 7848.421316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7848.421676] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7848.428267] Lustre: Skipped 2 previous similar messages [ 7848.434468] LustreError: Skipped 1 previous similar message [ 7853.540629] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7853.551857] LustreError: Skipped 1 previous similar message [ 7856.096105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7856.194287] Lustre: server umount lustre-OST0000 complete [ 7858.657431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.660891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7858.663660] Lustre: Skipped 2 previous similar messages [ 7858.670544] Lustre: Skipped 3 previous similar messages [ 7864.401474] Lustre: server umount lustre-MDT0000 complete [ 7866.464694] LustreError: 362651:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321448 with bad export cookie 13473624938628893547 [ 7866.467184] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7866.472443] LustreError: 362651:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7871.742783] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7873.133211] Key type lgssc unregistered [ 7873.254189] LNet: 368017:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7873.258259] LNet: Removed LNI 192.168.203.123@tcp [ 7873.679156] Key type .llcrypt unregistered [ 7873.680836] Key type ._llcrypt unregistered [ 7887.344294] alg: No test for adler32 (adler32-zlib) [ 7888.095409] Key type ._llcrypt registered [ 7888.096910] Key type .llcrypt registered [ 7888.140883] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7894.863814] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7895.293991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7895.483169] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 7895.572437] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7895.574884] LNet: Accept secure, port 988 [ 7897.160120] Key type lgssc registered [ 7897.569251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7901.240213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7904.296099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7907.141600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7910.015226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7913.184700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7913.212297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7914.321477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7914.338995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7914.381811] Lustre: lustre-MDT0000: new disk, initializing [ 7914.426340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7914.435553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7916.191869] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7921.279470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7921.320885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.370554] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7921.394371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7921.397865] Lustre: Skipped 1 previous similar message [ 7921.437942] Lustre: lustre-MDT0001: new disk, initializing [ 7921.483765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7921.502976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7921.508298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7923.395388] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7928.423100] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7931.238127] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7934.858237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7934.897447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7934.902081] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7935.050278] Lustre: lustre-OST0000: new disk, initializing [ 7935.053343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7935.104299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7937.455268] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 7942.676367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7942.683068] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7942.745930] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7945.274628] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7945.432465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7947.544858] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7947.652930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7952.865721] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7952.869228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7952.874384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7952.879161] Lustre: Skipped 1 previous similar message [ 7954.739374] Lustre: server umount lustre-OST0000 complete [ 7957.473545] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7957.477380] LustreError: Skipped 1 previous similar message [ 7957.479593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7957.488624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7957.498815] Lustre: Skipped 1 previous similar message [ 7962.638280] Lustre: server umount lustre-MDT0000 complete [ 7963.104829] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7963.111061] LustreError: Skipped 1 previous similar message [ 7964.407424] LustreError: 372957:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321546 with bad export cookie 8608274564649383376 [ 7964.408936] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7964.415843] LustreError: 372957:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7964.577347] Lustre: server umount lustre-MDT0001 complete [ 7968.700145] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 11:59:10 (1761321550) [ 7971.686665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7975.172557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7987.519658] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7991.522906] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7995.893116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7998.872244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8001.881859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8005.475597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8009.526678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8009.572179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8009.715259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8009.736808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8009.791252] Lustre: lustre-MDT0000: new disk, initializing [ 8009.858506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8009.871578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8012.554421] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8019.137532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8019.174525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8019.232576] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8019.257870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8019.261133] Lustre: Skipped 1 previous similar message [ 8019.306537] Lustre: lustre-MDT0001: new disk, initializing [ 8019.360066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8019.381900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8019.386666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8021.514161] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8027.093285] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8029.896632] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8033.420393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8033.456412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8033.461277] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8033.583473] Lustre: lustre-OST0000: new disk, initializing [ 8033.586147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8033.626706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.678411] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8038.922855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8038.927439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8041.256794] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8043.330140] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8043.443600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8045.420127] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8045.522318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8047.012124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8047.013577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8047.023336] Lustre: Skipped 2 previous similar messages [ 8047.028028] Lustre: Skipped 2 previous similar messages [ 8052.201298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8052.203841] Lustre: Skipped 1 previous similar message [ 8052.482693] Lustre: server umount lustre-OST0000 complete [ 8055.266112] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8055.270476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8057.314650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8057.317367] Lustre: Skipped 1 previous similar message [ 8060.433488] Lustre: server umount lustre-MDT0000 complete [ 8062.106301] LustreError: 380714:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321644 with bad export cookie 8608274564649384230 [ 8062.108687] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8062.112123] LustreError: 380714:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8066.179148] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 12:00:48 (1761321648) [ 8070.097464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8070.343848] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8070.388678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8072.373522] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8075.745969] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8076.202798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8078.373310] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8081.735379] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8084.305326] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8087.212393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8089.047521] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8092.466202] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8107.167096] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 8107.961826] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 8108.722290] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 8109.505678] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 8110.335368] Lustre: Failing over lustre-MDT0000 [ 8110.461628] Lustre: server umount lustre-MDT0000 complete [ 8110.463911] Lustre: Skipped 1 previous similar message [ 8111.075990] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8111.079349] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.081305] Lustre: Skipped 2 previous similar messages [ 8116.194159] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8116.201218] LustreError: Skipped 5 previous similar messages [ 8118.176169] Lustre: 370332:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761321693/real 1761321693] req@00000000ed49ceff x1846879430157952/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761321700 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8118.185944] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8124.385163] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8124.390453] LustreError: Skipped 10 previous similar messages [ 8124.479081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8134.628786] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0x7776bc1ba2f4fc7c to 0x7776bc1ba2f50057 [ 8134.634603] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 8134.750273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8134.752811] Lustre: Skipped 2 previous similar messages [ 8134.771909] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8136.045609] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8136.633606] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8139.750533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 8139.756959] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8139.776349] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 8142.160162] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8142.874696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8144.937352] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 8145.768778] Lustre: Failing over lustre-MDT0000 [ 8145.902947] Lustre: server umount lustre-MDT0000 complete [ 8146.278385] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8146.285085] LustreError: Skipped 9 previous similar messages [ 8149.987756] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8149.987758] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8149.987770] Lustre: Skipped 1 previous similar message [ 8157.153128] Lustre: 370333:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761321732/real 1761321732] req@00000000101998d9 x1846879430166784/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761321739 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8157.161629] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8159.879649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8163.299476] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0x7776bc1ba2f50057 to 0x7776bc1ba2f50359 [ 8163.307429] Lustre: MGC192.168.203.123@tcp: Connection restored to (at 0@lo) [ 8163.311086] Lustre: Skipped 2 previous similar messages [ 8163.439928] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8163.676437] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8165.145209] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8168.423237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 8168.433178] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8168.454794] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 8170.754933] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8171.445221] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8178.660503] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8178.661770] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.670065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8178.682568] Lustre: Skipped 2 previous similar messages [ 8178.685165] Lustre: Skipped 1 previous similar message [ 8180.545335] Lustre: server umount lustre-MDT0000 complete [ 8183.665598] LustreError: 389925:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321766 with bad export cookie 8608274564649386841 [ 8183.670196] LustreError: 389925:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8183.672426] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8183.778856] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8183.796581] LustreError: Skipped 19 previous similar messages [ 8206.055848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8206.621258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8206.623936] Lustre: Skipped 1 previous similar message [ 8210.750736] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8217.220991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.932884] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8224.108455] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8227.007941] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8230.229736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8232.269787] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8235.754750] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8238.568941] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 8243.682167] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8243.682660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8243.685669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8243.685677] Lustre: Skipped 2 previous similar messages [ 8243.687030] LustreError: Skipped 1 previous similar message [ 8243.693297] Lustre: Skipped 2 previous similar messages [ 8244.974094] Lustre: server umount lustre-OST0000 complete [ 8244.976377] Lustre: Skipped 2 previous similar messages [ 8253.922342] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.928595] LustreError: Skipped 4 previous similar messages [ 8254.530919] LustreError: 391134:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321836 with bad export cookie 8608274564649387548 [ 8254.535273] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8254.535796] LustreError: 391134:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8261.056596] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8262.484519] Key type lgssc unregistered [ 8262.660599] LNet: 394806:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8262.666987] LNet: Removed LNI 192.168.203.123@tcp [ 8263.172499] Key type .llcrypt unregistered [ 8263.173892] Key type ._llcrypt unregistered [ 8276.655323] alg: No test for adler32 (adler32-zlib) [ 8277.408566] Key type ._llcrypt registered [ 8277.410912] Key type .llcrypt registered [ 8277.481619] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8278.040989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8278.345994] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 8278.461897] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8278.465731] LNet: Accept secure, port 988 [ 8280.072161] Key type lgssc registered [ 8280.601504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8284.444788] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.447937] LustreError: 396290:0:(osd_handler.c:8221:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8284.463633] LustreError: 396290:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8284.467513] LustreError: 396290:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8284.471084] LustreError: 396290:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8284.474517] LustreError: 396290:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8287.682754] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8287.686809] LustreError: 396687:0:(osd_handler.c:8221:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8287.698209] LustreError: 396687:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8287.701562] LustreError: 396687:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8287.704144] LustreError: 396687:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8287.707944] LustreError: 396687:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8295.854488] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8297.270354] Key type lgssc unregistered [ 8297.424568] LNet: 397643:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8297.428787] LNet: Removed LNI 192.168.203.123@tcp [ 8297.897522] Key type .llcrypt unregistered [ 8297.899396] Key type ._llcrypt unregistered [ 8306.829561] alg: No test for adler32 (adler32-zlib) [ 8307.581363] Key type ._llcrypt registered [ 8307.583276] Key type .llcrypt registered [ 8307.643763] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8315.563227] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8315.971645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8316.224273] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 8316.319359] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8316.321932] LNet: Accept secure, port 988 [ 8317.928835] Key type lgssc registered [ 8318.443602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8323.325555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8326.593248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8330.031366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8333.746204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8338.023085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8338.071396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8339.223338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8339.248678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8339.298575] Lustre: lustre-MDT0000: new disk, initializing [ 8339.373214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8339.412182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8341.875894] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8348.743736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8348.795794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8348.851347] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8348.885200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8348.889047] Lustre: Skipped 1 previous similar message [ 8348.956803] Lustre: lustre-MDT0001: new disk, initializing [ 8349.038310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8349.065124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8349.072515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8351.649117] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8358.857981] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8362.712973] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8367.125120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8367.173796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8367.178067] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8367.352172] Lustre: lustre-OST0000: new disk, initializing [ 8367.354873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8367.428694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8370.115054] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8373.775137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8373.779391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8375.511878] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8377.738197] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8377.836462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8379.794359] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8379.910386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8383.969055] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8383.971360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8383.980163] LustreError: Skipped 1 previous similar message [ 8383.981256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8383.988311] Lustre: Skipped 1 previous similar message [ 8383.996009] Lustre: Skipped 1 previous similar message [ 8387.230777] Lustre: server umount lustre-OST0000 complete [ 8389.090205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8389.097155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8389.099722] Lustre: Skipped 1 previous similar message [ 8394.209936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8394.213586] Lustre: Skipped 1 previous similar message [ 8395.216439] Lustre: server umount lustre-MDT0000 complete [ 8397.366107] LustreError: 400912:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321979 with bad export cookie 17506443040400505870 [ 8397.367665] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8397.377245] LustreError: 400912:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8397.560875] Lustre: server umount lustre-MDT0001 complete [ 8401.978368] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8402.939255] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:06:24 (1761321984) [ 8406.748362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.030929] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8407.092431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.854449] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8412.132669] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8415.083662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8415.274040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8417.941425] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8422.569777] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8425.576816] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8429.085062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8429.241769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8431.339695] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8435.604372] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8439.167804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8439.208692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8439.214292] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 8439.285931] Lustre: lustre-OST0001: new disk, initializing [ 8439.289196] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 8441.446270] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8444.448685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8444.459386] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8446.645656] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8449.510048] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8449.510585] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8449.512632] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8449.514011] LustreError: Skipped 1 previous similar message [ 8449.520676] Lustre: Skipped 2 previous similar messages [ 8451.023831] Lustre: server umount lustre-OST0001 complete [ 8451.980942] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8454.627962] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8454.633597] LustreError: Skipped 1 previous similar message [ 8459.752331] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8459.759433] LustreError: Skipped 1 previous similar message [ 8464.868056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.869917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8464.873633] Lustre: Skipped 1 previous similar message [ 8464.877427] Lustre: Skipped 1 previous similar message [ 8469.987958] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8469.994772] LustreError: Skipped 3 previous similar messages [ 8470.206126] Lustre: server umount lustre-OST0000 complete [ 8475.108830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8475.110472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8475.114125] Lustre: Skipped 1 previous similar message [ 8475.116119] Lustre: Skipped 3 previous similar messages [ 8486.368241] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8486.458307] Lustre: server umount lustre-MDT0000 complete [ 8489.396103] LustreError: 405494:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322071 with bad export cookie 17506443040400506724 [ 8489.399214] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8489.407661] LustreError: 405494:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8503.049448] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8504.378906] Key type lgssc unregistered [ 8504.536618] LNet: 410413:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8504.542046] LNet: Removed LNI 192.168.203.123@tcp [ 8504.941789] Key type .llcrypt unregistered [ 8504.943377] Key type ._llcrypt unregistered [ 8524.454122] alg: No test for adler32 (adler32-zlib) [ 8525.205414] Key type ._llcrypt registered [ 8525.207049] Key type .llcrypt registered [ 8525.258890] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8532.401290] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8532.935608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8533.114300] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 8533.216824] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8533.219121] LNet: Accept secure, port 988 [ 8534.808213] Key type lgssc registered [ 8535.269672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8539.225055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8542.158532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8544.933687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8547.758979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8551.154100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8551.184453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8552.307402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8552.323942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8552.367422] Lustre: lustre-MDT0000: new disk, initializing [ 8552.405094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8552.415613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8554.329945] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8559.465319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8559.498378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8559.538279] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8559.567428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8559.570155] Lustre: Skipped 1 previous similar message [ 8559.609574] Lustre: lustre-MDT0001: new disk, initializing [ 8559.649339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8559.666145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8559.670772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8561.566787] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8567.101414] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8569.869437] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8572.727671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8572.765099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8572.769326] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8572.886694] Lustre: lustre-OST0000: new disk, initializing [ 8572.889659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8572.930099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8574.695986] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8579.394827] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8581.132171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8581.135685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8581.470813] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8581.576927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8583.657860] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8583.772940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8586.210408] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8586.211259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8586.214186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8586.214591] LustreError: Skipped 1 previous similar message [ 8586.219756] Lustre: Skipped 1 previous similar message [ 8590.770460] Lustre: server umount lustre-OST0000 complete [ 8595.425219] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8595.428703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8595.436359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8595.439267] Lustre: Skipped 1 previous similar message [ 8596.455433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8596.458153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8598.676503] Lustre: server umount lustre-MDT0000 complete [ 8600.534901] LustreError: 414497:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322182 with bad export cookie 4811399160394754828 [ 8600.535456] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8600.541345] LustreError: 414497:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8600.735439] Lustre: server umount lustre-MDT0001 complete [ 8605.480214] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 12:09:47 (1761322187) [ 8610.178396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8614.287336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8614.506837] Lustre: lustre-MDT0000: new disk, initializing [ 8614.526595] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8614.563041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8616.941477] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8620.002706] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8621.337515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8621.534908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8623.840129] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8627.436516] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8630.372254] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8631.781174] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8631.784519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8631.797596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8631.800052] Lustre: Skipped 1 previous similar message [ 8636.897682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8636.900645] Lustre: Skipped 3 previous similar messages [ 8637.386879] Lustre: server umount lustre-MDT0000 complete [ 8639.252431] LustreError: 419625:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322221 with bad export cookie 4811399160394755661 [ 8639.253991] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8639.257588] LustreError: 419625:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8644.404759] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:10:26 (1761322226) [ 8648.021582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8648.268763] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8648.316315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.724243] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8653.793869] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8655.906901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8658.306330] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8661.933908] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8664.442373] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8667.601802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8667.811361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8667.814574] Lustre: Skipped 1 previous similar message [ 8669.753082] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8673.025585] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8687.021844] Lustre: Permanently deactivating lustre-OST0000 [ 8687.028850] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 8695.303216] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8696.731830] LustreError: 425405:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 8696.737458] LustreError: 425405:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8696.741777] LustreError: 425405:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8697.848970] LustreError: 425453:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8697.852265] LustreError: 425453:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8704.960078] Lustre: server umount lustre-OST0000 complete [ 8704.962078] Lustre: Skipped 1 previous similar message [ 8706.527899] LustreError: 425704:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8706.531477] LustreError: 425704:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8710.113473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8710.120237] Lustre: Skipped 1 previous similar message [ 8710.123131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8713.696166] Lustre: 425801:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322290/real 1761322290] req@00000000f5030eee x1846880098098368/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761322296 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8713.818913] Lustre: server umount lustre-MDT0000 complete [ 8715.233173] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8715.244018] LustreError: Skipped 1 previous similar message [ 8715.689102] LustreError: 422222:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322298 with bad export cookie 4811399160394756221 [ 8715.690685] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8715.693788] LustreError: 422222:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8721.888336] Lustre: 426003:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322298/real 1761322298] req@0000000052fde74c x1846880098099328/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761322304 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8726.850316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.933956] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8730.245597] LustreError: 426912:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 8733.140598] Lustre: 427257:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.123@tcp [ 8740.320487] Lustre: 427356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322316/real 1761322316] req@0000000085aaaffc x1846880098099712/t0(0) o251->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322322 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8740.349586] Lustre: server umount MGS complete [ 8740.351932] Lustre: Skipped 1 previous similar message [ 8745.088857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8745.292104] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8745.307132] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8745.310347] Lustre: Skipped 1 previous similar message [ 8745.340456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8747.102328] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8751.115552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.290070] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8753.427456] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8757.111929] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8759.652865] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8760.542565] Lustre: Permanently reactivating lustre-OST0000 [ 8760.550539] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 8760.553779] Lustre: Skipped 2 previous similar messages [ 8774.969241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8776.872350] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8779.904962] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8796.130861] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8796.133799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8796.138661] Lustre: Skipped 1 previous similar message [ 8796.141053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8796.143619] Lustre: Skipped 1 previous similar message [ 8806.368131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8806.488932] Lustre: server umount lustre-OST0000 complete [ 8816.355551] LustreError: 427964:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322398 with bad export cookie 4811399160394757173 [ 8816.356482] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8816.359872] LustreError: 427964:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8822.612884] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8823.847498] Key type lgssc unregistered [ 8823.981460] LNet: 432200:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8823.985858] LNet: Removed LNI 192.168.203.123@tcp [ 8824.366692] Key type .llcrypt unregistered [ 8824.368927] Key type ._llcrypt unregistered [ 8832.390754] alg: No test for adler32 (adler32-zlib) [ 8833.142693] Key type ._llcrypt registered [ 8833.144396] Key type .llcrypt registered [ 8833.192281] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8840.346972] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8840.848044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8841.009479] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 8841.128682] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8841.131265] LNet: Accept secure, port 988 [ 8842.760143] Key type lgssc registered [ 8843.259297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8847.676381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8850.962067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8853.873930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8856.799676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8860.244272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8860.296370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8861.422723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8861.437521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8861.480979] Lustre: lustre-MDT0000: new disk, initializing [ 8861.523533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8861.533544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8863.607047] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8869.636025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8869.673943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8869.724825] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8869.758118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8869.761853] Lustre: Skipped 1 previous similar message [ 8869.809679] Lustre: lustre-MDT0001: new disk, initializing [ 8869.867706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8869.886856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8869.896951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8872.167152] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8877.470213] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8880.293319] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8883.502473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8883.538982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8883.542985] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8883.672700] Lustre: lustre-OST0000: new disk, initializing [ 8883.676708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8883.732079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8884.439117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8884.443719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8886.147123] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8891.536065] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8893.845940] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8893.973913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8896.484435] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8896.610074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8900.066962] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8900.067780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.070606] LustreError: Skipped 1 previous similar message [ 8900.077212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8900.077632] Lustre: Skipped 1 previous similar message [ 8905.187213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8905.191232] Lustre: Skipped 2 previous similar messages [ 8910.306640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8910.309399] Lustre: Skipped 1 previous similar message [ 8911.840214] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8911.967605] Lustre: server umount lustre-OST0000 complete [ 8915.425590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8915.426198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8915.430797] Lustre: Skipped 1 previous similar message [ 8915.435458] Lustre: Skipped 1 previous similar message [ 8919.823284] Lustre: server umount lustre-MDT0000 complete [ 8920.547318] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8920.555453] LustreError: Skipped 1 previous similar message [ 8921.578212] LustreError: 435465:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322504 with bad export cookie 8122068722995549419 [ 8921.579976] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8921.585207] LustreError: 435465:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8921.733126] Lustre: server umount lustre-MDT0001 complete [ 8925.391918] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8926.157380] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:15:08 (1761322508) [ 8929.275320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8929.514235] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8929.560734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8931.610557] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8934.881956] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8935.780891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8935.945965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8937.866341] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8941.207205] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8943.632571] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8946.918193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8947.093920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8948.975544] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 8952.693764] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8953.552791] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8964.579594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8970.724170] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8970.724690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8970.730361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8970.735401] Lustre: Skipped 1 previous similar message [ 8972.922929] Lustre: server umount lustre-OST0000 complete [ 8975.842402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8975.849014] Lustre: Skipped 1 previous similar message [ 8980.773448] Lustre: server umount lustre-MDT0000 complete [ 8980.962619] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8980.968688] LustreError: Skipped 1 previous similar message [ 8982.718992] LustreError: 440046:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322565 with bad export cookie 8122068722995550280 [ 8982.721565] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8982.726247] LustreError: 440046:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8989.047705] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8990.369654] Key type lgssc unregistered [ 8990.526180] LNet: 443930:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8990.531551] LNet: Removed LNI 192.168.203.123@tcp [ 8990.970643] Key type .llcrypt unregistered [ 8990.972403] Key type ._llcrypt unregistered [ 9002.577242] alg: No test for adler32 (adler32-zlib) [ 9003.329376] Key type ._llcrypt registered [ 9003.330673] Key type .llcrypt registered [ 9003.379679] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9003.486667] Key type .llcrypt unregistered [ 9003.488149] Key type ._llcrypt unregistered [ 9011.387033] alg: No test for adler32 (adler32-zlib) [ 9012.139367] Key type ._llcrypt registered [ 9012.140944] Key type .llcrypt registered [ 9012.192946] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9012.585124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9012.854788] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 9012.955161] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 9012.958128] LNet: Accept secure, port 988 [ 9014.560153] Key type lgssc registered [ 9015.041697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9018.500820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9019.724121] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9019.762073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9021.541760] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9024.983551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9024.993673] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9025.140183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9026.780427] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9029.840585] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9030.626946] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9030.633365] LustreError: Skipped 3 previous similar messages [ 9032.888951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9033.053929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9034.803192] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9035.111522] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 9038.305049] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9038.308585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9038.314750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9040.354460] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9040.354859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9040.359209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9044.263325] Lustre: server umount lustre-OST0000 complete [ 9048.545570] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9048.549274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9048.555481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9048.558660] Lustre: Skipped 1 previous similar message [ 9050.593973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9050.595054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9051.803806] Lustre: server umount lustre-MDT0000 complete [ 9053.320171] LustreError: 448715:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322635 with bad export cookie 18190941875111440871 [ 9053.322378] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9053.324556] LustreError: 448715:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9053.485673] Lustre: server umount lustre-MDT0001 complete [ 9059.270496] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9060.540845] Key type lgssc unregistered [ 9060.678712] LNet: 449517:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9060.683135] LNet: Removed LNI 192.168.203.123@tcp [ 9061.028582] Key type .llcrypt unregistered [ 9061.030133] Key type ._llcrypt unregistered [ 9071.842341] alg: No test for adler32 (adler32-zlib) [ 9072.594424] Key type ._llcrypt registered [ 9072.595873] Key type .llcrypt registered [ 9072.651337] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9073.146783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9073.346564] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 9073.452885] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 9073.455313] LNet: Accept secure, port 988 [ 9075.056150] Key type lgssc registered [ 9075.598723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9079.113976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9096.608584] LustreError: 450950:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 1: rc = -5 [ 9127.328328] LustreError: 450950:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 9157.024197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9158.713124] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9161.754993] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9164.440172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9179.622138] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.654922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9179.683468] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 9181.456908] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9184.738479] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9185.668114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9185.844110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9187.532926] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9190.704051] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9193.103327] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9199.012188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9199.013441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9199.018453] Lustre: Skipped 1 previous similar message [ 9202.281055] Lustre: server umount lustre-OST0000 complete [ 9204.194391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9204.195427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9204.199886] Lustre: Skipped 1 previous similar message [ 9204.202746] Lustre: Skipped 1 previous similar message [ 9205.838327] Lustre: server umount lustre-MDT0000 complete [ 9207.367247] LustreError: 453548:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322789 with bad export cookie 15083318787927967789 [ 9207.369385] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9207.372217] LustreError: 453548:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9207.526041] Lustre: server umount lustre-MDT0001 complete [ 9212.654765] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9213.760412] Key type lgssc unregistered [ 9213.883667] LNet: 454643:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9213.886937] LNet: Removed LNI 192.168.203.123@tcp [ 9214.193382] Key type .llcrypt unregistered [ 9214.195195] Key type ._llcrypt unregistered [ 9228.334904] alg: No test for adler32 (adler32-zlib) [ 9229.090439] Key type ._llcrypt registered [ 9229.091922] Key type .llcrypt registered [ 9229.252875] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9230.165592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9230.905153] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 9231.194370] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 9231.198505] LNet: Accept secure, port 988 [ 9232.920762] Key type lgssc registered [ 9234.925733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9246.539399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9248.357149] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9248.429315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9253.732542] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9253.859076] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9258.992343] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9262.888907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9263.240247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9267.146121] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9275.631992] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9280.941978] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9285.584356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9285.786764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9288.369234] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9294.073717] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9299.116055] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 9304.034132] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9304.038268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9304.044515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9304.548625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9304.551768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9304.564599] Lustre: Skipped 1 previous similar message [ 9307.374139] Lustre: server umount lustre-MDT0000 complete [ 9308.989525] LustreError: 456099:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.23@tcp arrived at 1761322891 with bad export cookie 9544118734830483961 [ 9308.999294] LustreError: 456099:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9309.670294] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9309.677100] LustreError: Skipped 2 previous similar messages [ 9310.208806] LustreError: 456099:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322892 with bad export cookie 9544118734830483296 [ 9310.211364] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9310.217029] LustreError: 456099:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9313.762047] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9313.765145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9313.779441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9313.782202] Lustre: Skipped 2 previous similar messages [ 9314.785519] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9314.796765] LustreError: Skipped 1 previous similar message [ 9315.811208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9315.819105] Lustre: Skipped 1 previous similar message [ 9316.410858] Lustre: server umount lustre-OST0000 complete [ 9321.038798] Lustre: server umount lustre-MDT0001 complete [ 9328.519196] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9330.264855] Key type lgssc unregistered [ 9330.468061] LNet: 459924:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9330.474408] LNet: Removed LNI 192.168.203.123@tcp [ 9331.052378] Key type .llcrypt unregistered [ 9331.054842] Key type ._llcrypt unregistered [ 9346.268431] alg: No test for adler32 (adler32-zlib) [ 9347.022369] Key type ._llcrypt registered [ 9347.024056] Key type .llcrypt registered [ 9347.079608] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9347.629710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9347.846661] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 9347.951805] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 9347.954716] LNet: Accept secure, port 988 [ 9349.568134] Key type lgssc registered [ 9350.303728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9354.923302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9356.176526] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9356.236291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9358.867536] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9361.382525] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9364.185407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9364.449610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9366.650707] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9370.942853] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9374.414762] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9378.500542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9378.681773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9381.358322] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9385.846194] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [ 9386.200965] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9390.054490] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9390.058114] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9390.064351] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9391.083574] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9391.084267] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9393.002439] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 9393.005160] Lustre: Skipped 1 previous similar message [ 9395.161143] Lustre: server umount lustre-MDT0001 complete [ 9396.198795] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9396.212052] LustreError: Skipped 1 previous similar message [ 9401.315932] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9401.315941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9401.316072] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9401.318689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9404.649249] Lustre: server umount lustre-OST0000 complete [ 9406.437633] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9414.116233] Lustre: 464443:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322990/real 1761322990] req@000000003860c180 x1846880952661632/t0(0) o251->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322996 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9414.230047] Lustre: server umount lustre-MDT0000 complete [ 9423.417365] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9424.905587] Key type lgssc unregistered [ 9425.109325] LNet: 465198:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9425.115438] LNet: Removed LNI 192.168.203.123@tcp [ 9425.545732] Key type .llcrypt unregistered [ 9425.547860] Key type ._llcrypt unregistered [ 9440.259241] alg: No test for adler32 (adler32-zlib) [ 9441.012516] Key type ._llcrypt registered [ 9441.014996] Key type .llcrypt registered [ 9441.095734] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 9448.558387] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9448.938849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9449.208486] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 9449.304023] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 9449.306485] LNet: Accept secure, port 988 [ 9450.904128] Key type lgssc registered [ 9451.334422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9455.164834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9458.166457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9461.167488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9463.902163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9467.768586] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9471.914136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9474.896626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9474.927484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9476.053599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9476.072945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9476.117036] Lustre: lustre-MDT0000: new disk, initializing [ 9476.169174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9476.184259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9478.367413] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9483.560436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9487.553424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9487.592789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9487.602772] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9487.735134] Lustre: lustre-OST0000: new disk, initializing [ 9487.738571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9487.742105] Lustre: Skipped 1 previous similar message [ 9487.783624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9489.990454] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9495.549666] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9500.044662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9503.644917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9503.691537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9503.757965] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9503.797400] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9503.840541] Lustre: lustre-MDT0001: new disk, initializing [ 9503.875406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9503.880033] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9503.908475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9506.129784] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9511.320767] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 9511.437684] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9513.384676] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 9514.525623] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 9519.075794] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9519.081164] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9519.086925] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9519.089445] Lustre: Skipped 1 previous similar message [ 9520.492851] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 9524.194596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9524.198760] Lustre: Skipped 1 previous similar message [ 9529.322632] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9529.325440] Lustre: Skipped 2 previous similar messages [ 9531.363339] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9531.435107] Lustre: server umount lustre-MDT0001 complete [ 9534.434104] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9534.440163] LustreError: Skipped 1 previous similar message [ 9539.556525] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9539.566618] LustreError: Skipped 1 previous similar message [ 9544.674675] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9544.681547] LustreError: Skipped 1 previous similar message [ 9549.281187] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9549.285351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9549.290769] Lustre: Skipped 1 previous similar message [ 9549.293344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9549.296238] Lustre: Skipped 1 previous similar message [ 9549.794896] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9551.359048] Lustre: server umount lustre-OST0000 complete [ 9564.640813] Lustre: 473613:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323141/real 1761323141] req@00000000bfeaae71 x1846881058570944/t0(0) o251->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323147 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9564.714737] Lustre: server umount lustre-MDT0000 complete [ 9571.403292] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9572.563815] Key type lgssc unregistered [ 9572.699441] LNet: 474368:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9572.704593] LNet: Removed LNI 192.168.203.123@tcp [ 9573.056162] Key type .llcrypt unregistered [ 9573.057868] Key type ._llcrypt unregistered [ 9589.180383] alg: No test for adler32 (adler32-zlib) [ 9589.932330] Key type ._llcrypt registered [ 9589.933561] Key type .llcrypt registered [ 9589.979608] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9590.098197] Key type .llcrypt unregistered [ 9590.099615] Key type ._llcrypt unregistered [ 9597.344757] alg: No test for adler32 (adler32-zlib) [ 9598.096366] Key type ._llcrypt registered [ 9598.097687] Key type .llcrypt registered [ 9598.150061] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9598.521057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9598.768710] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 9598.855820] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 9598.858367] LNet: Accept secure, port 988 [ 9600.448131] Key type lgssc registered [ 9600.844906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9603.988478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9606.592909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9609.084979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9611.981147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9612.024084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9613.141804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9613.158662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9613.200668] Lustre: lustre-MDT0000: new disk, initializing [ 9613.251481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9613.262171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9615.287688] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9620.696909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9620.737982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9620.785123] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9620.807268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9620.810854] Lustre: Skipped 1 previous similar message [ 9620.856794] Lustre: lustre-MDT0001: new disk, initializing [ 9620.908262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9620.924686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9620.931074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9622.706148] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9627.228835] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9629.657802] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9633.090158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9633.130827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9633.135125] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9633.270979] Lustre: lustre-OST0000: new disk, initializing [ 9633.273921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9633.309746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9635.187213] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9639.557452] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9642.002424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9642.008699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9651.681459] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9651.685378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9651.691828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9652.193739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9652.194774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9652.199300] Lustre: Skipped 1 previous similar message [ 9652.208324] Lustre: Skipped 1 previous similar message [ 9654.097498] Lustre: server umount lustre-MDT0000 complete [ 9655.654287] LustreError: 477850:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323238 with bad export cookie 9584925348434014759 [ 9655.655595] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9655.659518] LustreError: 477850:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9655.832661] Lustre: server umount lustre-MDT0001 complete [ 9663.456157] Lustre: 481105:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323239/real 1761323239] req@0000000099276e4d x1846881215855680/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761323245 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9663.551302] Lustre: server umount lustre-OST0000 complete [ 9667.240629] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 12:27:29 (1761323249) [ 9667.622813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9670.016759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9670.182067] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9670.208359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9671.751937] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9674.955572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9675.087833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9676.762587] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9679.663950] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9682.118310] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9684.663728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9684.695560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9684.787846] Lustre: Found index 0 for lustre-OST0000, updating log [ 9684.850322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9685.864575] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [ 9686.646098] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9689.672276] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9695.203537] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9695.203820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9695.207460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9695.212129] Lustre: Skipped 1 previous similar message [ 9695.214811] Lustre: Skipped 1 previous similar message [ 9697.257513] Lustre: server umount lustre-OST0000 complete [ 9700.322577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9700.323176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9700.329099] Lustre: Skipped 1 previous similar message [ 9700.333728] Lustre: Skipped 2 previous similar messages [ 9704.353696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9704.355984] Lustre: Skipped 1 previous similar message [ 9706.443601] Lustre: server umount lustre-MDT0000 complete [ 9707.763809] LustreError: 481912:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323290 with bad export cookie 9584925348434016082 [ 9707.765972] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9707.767734] LustreError: 481912:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9710.880364] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 12:28:12 (1761323292) [ 9713.436865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9715.723110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9718.172047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9720.110781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9726.428327] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 9729.881482] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9733.584784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9736.132823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9738.396723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9741.156531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9744.440859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9744.477402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9744.571921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9744.584304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9744.623742] Lustre: lustre-MDT0000: new disk, initializing [ 9744.653345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9744.661690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9746.448193] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9751.555602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9751.594229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9751.635163] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9751.655445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9751.658679] Lustre: Skipped 1 previous similar message [ 9751.698378] Lustre: lustre-MDT0001: new disk, initializing [ 9751.750226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9751.755524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9753.576320] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9758.336440] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9760.799924] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9763.642280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9763.678306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9763.682744] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9763.789954] Lustre: lustre-OST0000: new disk, initializing [ 9763.792680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9765.591876] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 9770.243617] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9772.058214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9772.063815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9772.381971] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9772.474389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9774.512465] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9774.613516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9777.124812] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9777.126138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9777.129079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9777.138291] Lustre: Skipped 1 previous similar message [ 9777.144630] Lustre: Skipped 1 previous similar message [ 9781.677306] Lustre: server umount lustre-OST0000 complete [ 9781.679284] Lustre: Skipped 1 previous similar message [ 9787.362281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9787.367368] Lustre: Skipped 1 previous similar message [ 9791.320525] LustreError: 488939:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323373 with bad export cookie 9584925348434017132 [ 9791.323397] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9791.325580] LustreError: 488939:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9802.725182] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 9806.788293] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9811.504921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9814.811185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9817.386494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9820.190236] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9820.990639] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9649 sec ========== 12:30:02 (1761323402) [ 9838.699457] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9839.850716] Key type lgssc unregistered [ 9839.977293] LNet: 497669:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9839.980484] LNet: Removed LNI 192.168.203.123@tcp [ 9840.332051] Key type .llcrypt unregistered [ 9840.333592] Key type ._llcrypt unregistered