[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 619922708 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002151] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006859] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009119] LSM: Security Framework initializing [ 0.010052] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.012073] *** VALIDATE selinux *** [ 0.018637] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023108] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024191] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026135] *** VALIDATE tmpfs *** [ 0.028250] *** VALIDATE proc *** [ 0.029305] *** VALIDATE cgroup *** [ 0.030022] *** VALIDATE cgroup2 *** [ 0.032055] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034100] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035021] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036049] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041505] debug: unmapping init [mem 0xffffffffbd059000-0xffffffffbd060fff] [ 0.043193] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044630] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045021] ... version: 2 [ 0.046014] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049009] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052279] rcu: Hierarchical SRCU implementation. [ 0.054382] smp: Bringing up secondary CPUs ... [ 0.055705] x86: Booting SMP configuration: [ 0.056023] .... node #0, CPUs: #1 #2 #3 [ 0.076015] smp: Brought up 1 node, 4 CPUs [ 0.078019] smpboot: Max logical packages: 1 [ 0.079015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.122617] node 0 deferred pages initialised in 41ms [ 0.129097] devtmpfs: initialized [ 0.130000] x86/mm: Memory block size: 128MB [ 0.132805] gcov: version magic: 0x41383552 [ 0.139260] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145285] pinctrl core: initialized pinctrl subsystem [ 0.148201] [ 0.148732] ************************************************************* [ 0.152016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154011] ** ** [ 0.159014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161017] ** ** [ 0.162000] ** This means that this kernel is built to expose internal ** [ 0.162000] ** IOMMU data structures, which may compromise security on ** [ 0.166019] ** your system. ** [ 0.169014] ** ** [ 0.171014] ** If you see this message and you are not debugging the ** [ 0.175015] ** kernel, report this immediately to your vendor! ** [ 0.178014] ** ** [ 0.179000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179000] ************************************************************* [ 0.180000] NET: Registered protocol family 16 [ 0.183000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.195043] cpuidle: using governor menu [ 0.204600] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205000] PCI: Using configuration type 1 for base access [ 0.206128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.221182] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224284] cryptd: max_cpu_qlen set to 1000 [ 0.228299] ACPI: Added _OSI(Module Device) [ 0.229528] ACPI: Added _OSI(Processor Device) [ 0.231015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243640] ACPI: Interpreter enabled [ 0.245086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246016] ACPI: Using IOAPIC for interrupt routing [ 0.249116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.251000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.264343] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.280083] acpiphp: Slot [2] registered [ 0.281400] acpiphp: Slot [3] registered [ 0.283162] acpiphp: Slot [4] registered [ 0.285090] acpiphp: Slot [5] registered [ 0.287142] acpiphp: Slot [6] registered [ 0.288115] acpiphp: Slot [7] registered [ 0.290114] acpiphp: Slot [8] registered [ 0.291144] acpiphp: Slot [9] registered [ 0.293133] acpiphp: Slot [10] registered [ 0.295122] acpiphp: Slot [11] registered [ 0.298120] acpiphp: Slot [12] registered [ 0.299236] acpiphp: Slot [13] registered [ 0.302112] acpiphp: Slot [14] registered [ 0.304116] acpiphp: Slot [15] registered [ 0.306164] acpiphp: Slot [16] registered [ 0.308191] acpiphp: Slot [17] registered [ 0.311105] acpiphp: Slot [18] registered [ 0.312072] acpiphp: Slot [19] registered [ 0.314140] acpiphp: Slot [20] registered [ 0.315000] acpiphp: Slot [21] registered [ 0.317135] acpiphp: Slot [22] registered [ 0.318107] acpiphp: Slot [23] registered [ 0.320142] acpiphp: Slot [24] registered [ 0.322134] acpiphp: Slot [25] registered [ 0.323743] acpiphp: Slot [26] registered [ 0.326145] acpiphp: Slot [27] registered [ 0.328126] acpiphp: Slot [28] registered [ 0.330167] acpiphp: Slot [29] registered [ 0.332097] acpiphp: Slot [30] registered [ 0.333107] acpiphp: Slot [31] registered [ 0.334000] PCI host bridge to bus 0000:00 [ 0.334027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.337021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.343043] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.346113] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.349074] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.352143] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.355280] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.359425] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.375039] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.381016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.384000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.386725] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.389033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.391558] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395008] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.406888] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.410937] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.415000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.425021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.436020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.459023] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.466035] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.477020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.487019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.514021] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.536458] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.552017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.567018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.599028] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.614115] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.630014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.645016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.667015] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.679000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.685018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.693017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.713016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.728864] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.739025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.748016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.763016] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.776154] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.778347] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.783405] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.792396] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.798316] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.809042] iommu: Default domain type: Passthrough [ 0.814412] SCSI subsystem initialized [ 0.817114] ACPI: bus type USB registered [ 0.821105] usbcore: registered new interface driver usbfs [ 0.828092] usbcore: registered new interface driver hub [ 0.835111] usbcore: registered new device driver usb [ 0.842172] pps_core: LinuxPPS API ver. 1 registered [ 0.848013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.853069] PTP clock support registered [ 0.862079] EDAC MC: Ver: 3.0.0 [ 0.866098] PCI: Using ACPI for IRQ routing [ 0.870234] NetLabel: Initializing [ 0.872018] NetLabel: domain hash size = 128 [ 0.873021] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.874194] NetLabel: unlabeled traffic allowed by default [ 0.877295] vgaarb: loaded [ 0.880268] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.882024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.891736] clocksource: Switched to clocksource kvm-clock [ 1.099615] VFS: Disk quotas dquot_6.6.0 [ 1.103345] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.110291] *** VALIDATE ramfs *** [ 1.112991] *** VALIDATE hugetlbfs *** [ 1.119015] pnp: PnP ACPI init [ 1.121283] pnp: PnP ACPI: found 6 devices [ 1.170386] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.174647] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.176873] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.179026] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.182824] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.185797] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.189162] NET: Registered protocol family 2 [ 1.196564] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.202475] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.207035] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.212694] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.215731] TCP: Hash tables configured (established 65536 bind 65536) [ 1.219182] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.222329] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.225667] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.228595] NET: Registered protocol family 1 [ 1.235408] RPC: Registered named UNIX socket transport module. [ 1.238216] RPC: Registered udp transport module. [ 1.239716] RPC: Registered tcp transport module. [ 1.241877] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.245473] NET: Registered protocol family 44 [ 1.247920] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.249822] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.254436] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.257425] PCI: CLS 0 bytes, default 64 [ 1.260419] Unpacking initramfs... [ 3.820359] debug: unmapping init [mem 0xffff8a5fbcc54000-0xffff8a5fbffbffff] [ 3.824961] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.827705] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.831106] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.505843] Initialise system trusted keyrings [ 4.507159] Key type blacklist registered [ 4.510045] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.518918] zbud: loaded [ 4.522568] *** VALIDATE nfs *** [ 4.523505] *** VALIDATE nfs4 *** [ 4.524608] pstore: using deflate compression [ 4.528189] Platform Keyring initialized [ 4.687443] NET: Registered protocol family 38 [ 4.689228] Key type asymmetric registered [ 4.690731] Asymmetric key parser 'x509' registered [ 4.696198] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.704122] io scheduler mq-deadline registered [ 4.705745] io scheduler kyber registered [ 4.707797] io scheduler bfq registered [ 4.709799] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.714545] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.720643] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.726932] ACPI: Power Button [PWRF] [ 4.966238] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.181283] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.684230] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.937962] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.370548] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.440714] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.493893] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.502611] Non-volatile memory driver v1.3 [ 6.504642] Linux agpgart interface v0.103 [ 6.542917] virtio_blk virtio1: [vda] 67992 512-byte logical blocks (34.8 MB/33.2 MiB) [ 6.545616] vda: detected capacity change from 0 to 34811904 [ 6.577027] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.580351] vdb: detected capacity change from 0 to 1073741824 [ 6.622103] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.628154] vdc: detected capacity change from 0 to 2621440000 [ 6.676402] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.690033] vdd: detected capacity change from 0 to 2621440000 [ 6.730916] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.745242] vde: detected capacity change from 0 to 4294967296 [ 6.789193] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.791575] vdf: detected capacity change from 0 to 4294967296 [ 6.803637] libphy: Fixed MDIO Bus: probed [ 6.814540] usbcore: registered new interface driver usbserial_generic [ 6.821437] usbserial: USB Serial support registered for generic [ 6.832655] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.846964] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.848302] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.854503] mousedev: PS/2 mouse device common for all mice [ 6.857970] rtc_cmos 00:05: RTC can wake from S4 [ 6.868925] rtc_cmos 00:05: registered as rtc0 [ 6.870466] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.882081] intel_pstate: CPU model not supported [ 6.894193] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.898637] hid: raw HID events driver (C) Jiri Kosina [ 6.913516] usbcore: registered new interface driver usbhid [ 6.918902] usbhid: USB HID core driver [ 6.928880] drop_monitor: Initializing network drop monitor service [ 6.930263] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.949724] Initializing XFRM netlink socket [ 6.951961] NET: Registered protocol family 10 [ 6.978214] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.979791] Segment Routing with IPv6 [ 6.982498] NET: Registered protocol family 17 [ 6.989320] mpls_gso: MPLS GSO support [ 7.050641] RAS: Correctable Errors collector initialized. [ 7.056176] AVX version of gcm_enc/dec engaged. [ 7.058720] AES CTR mode by8 optimization enabled [ 7.270233] sched_clock: Marking stable (7270210220, 0)->(8678148999, -1407938779) [ 7.274732] registered taskstats version 1 [ 7.278139] Loading compiled-in X.509 certificates [ 7.289716] zswap: loaded using pool lzo/zbud [ 7.365076] Key type big_key registered [ 7.405823] Key type encrypted registered [ 7.407010] ima: No TPM chip found, activating TPM-bypass! [ 7.408602] ima: Allocated hash algorithm: sha1 [ 7.413120] ima: No architecture policies found [ 7.414505] evm: Initialising EVM extended attributes: [ 7.416311] evm: security.selinux [ 7.417528] evm: security.ima [ 7.422787] evm: security.capability [ 7.425342] evm: HMAC attrs: 0x1 [ 7.472143] rtc_cmos 00:05: setting system clock to 2025-10-24 16:02:54 UTC (1761321774) [ 7.479548] debug: unmapping init [mem 0xffffffffbe003000-0xffffffffbe1fffff] [ 7.493142] debug: unmapping init [mem 0xffffffffbcd82000-0xffffffffbd058fff] [ 7.514067] Write protecting the kernel read-only data: 28672k [ 7.517686] debug: unmapping init [mem 0xffffffffbb403000-0xffffffffbb5fffff] [ 7.537420] debug: unmapping init [mem 0xffffffffbbd14000-0xffffffffbbdfffff] [ 7.616765] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.641707] systemd[1]: Detected virtualization kvm. [ 7.650305] systemd[1]: Detected architecture x86-64. [ 7.660729] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.707708] systemd[1]: No hostname configured. [ 7.723385] systemd[1]: Set hostname to . [ 7.725016] random: systemd: uninitialized urandom read (16 bytes read) [ 7.727127] systemd[1]: Initializing machine ID from random generator. [ 8.213790] random: systemd: uninitialized urandom read (16 bytes read) [ 8.215914] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 8.220395] random: systemd: uninitialized urandom read (16 bytes read) [ 8.238452] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 8.262757] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.099802] device-mapper: uevent: version 1.0.3 [ 10.166507] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 12.174619] virtio_net virtio0 ens2: renamed from eth0 [ 12.260402] random: fast init done [ 12.628236] scsi host0: ata_piix [ 12.705840] scsi host1: ata_piix [ 12.707176] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.709547] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.031986] random: crng init done [ 17.039332] random: 7 urandom warning(s) missed due to ratelimiting [ 20.296961] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.040836] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.572676] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.470991] SELinux: Disabled at runtime. [ 25.633033] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.689314] systemd[1]: Detected virtualization kvm. [ 25.691270] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.680768] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.696666] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.712995] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.730642] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.734949] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.779525] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.838977] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 28.039334] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 29.464715] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.391368] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.472962] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.065825] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.439830] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 35.616751] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 36.886180] NFS: Registering the id_resolver key type [ 36.888059] Key type id_resolver registered [ 36.889524] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg423-server login: [ 75.395670] libcfs: loading out-of-tree module taints kernel. [ 75.428672] alg: No test for adler32 (adler32-zlib) [ 76.179424] Key type ._llcrypt registered [ 76.180946] Key type .llcrypt registered [ 76.234255] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 86.714867] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 87.456518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 87.798836] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 88.140311] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 88.144147] LNet: Accept secure, port 988 [ 89.783341] Key type lgssc registered [ 90.615864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 101.370878] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 102.841373] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.426885] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.211587] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 118.498443] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 127.545830] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 134.915829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 134.973506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 136.132609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 136.147700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 136.191704] Lustre: lustre-MDT0000: new disk, initializing [ 136.234711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 136.251981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 138.505792] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 146.504692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 146.553723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.593079] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 146.609548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 146.612564] Lustre: Skipped 1 previous similar message [ 146.663909] Lustre: lustre-MDT0001: new disk, initializing [ 146.706788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 146.731134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 146.737156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 148.936931] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 155.790978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 155.830766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 155.834840] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 155.950312] Lustre: lustre-OST0000: new disk, initializing [ 155.952885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 155.983192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.967288] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 161.265063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 161.269741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 164.164831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 164.204908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 164.211836] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 164.272822] Lustre: lustre-OST0001: new disk, initializing [ 164.275799] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 164.305492] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.591791] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 168.970769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 168.981984] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 174.527936] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 180.719879] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 188.178806] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing check_logdir /tmp/testlogs/ [ 191.269891] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing yml_node [ 194.225229] Lustre: DEBUG MARKER: Client: 2.15.7.12 [ 195.894220] Lustre: DEBUG MARKER: MDS: 2.15.7.12 [ 197.647320] Lustre: DEBUG MARKER: OSS: 2.15.7.12 [ 198.844834] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 12:06:05 EDT 2025 [ 203.826324] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 204.794455] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 218.593953] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 218.598472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 218.607425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 220.128697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 220.128871] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 223.904068] Lustre: server umount lustre-MDT0000 complete [ 225.248463] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 225.258220] LustreError: Skipped 3 previous similar messages [ 226.147394] LustreError: 6231:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321993 with bad export cookie 17165493054421827207 [ 226.148816] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 226.152779] LustreError: 6231:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 230.367947] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.370026] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 230.370507] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 230.370513] Lustre: Skipped 3 previous similar messages [ 230.379633] Lustre: Skipped 3 previous similar messages [ 230.394461] LustreError: Skipped 1 previous similar message [ 231.392119] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 231.398015] LustreError: Skipped 1 previous similar message [ 232.448310] Lustre: server umount lustre-MDT0001 complete [ 234.376749] Lustre: server umount lustre-OST0000 complete [ 239.376708] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 242.866805] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 247.331883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 250.620440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 253.496406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 256.463897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 259.456781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 259.488068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 259.584508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 259.596882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 259.637921] Lustre: lustre-MDT0000: new disk, initializing [ 259.667040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.677815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 261.478718] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 266.716525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 266.755645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 266.798067] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 266.967474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 268.820571] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 273.819236] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 276.522690] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 279.499907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 279.535909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 279.539922] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 279.653829] Lustre: lustre-OST0000: new disk, initializing [ 279.656088] Lustre: Skipped 1 previous similar message [ 279.658736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 279.662399] Lustre: Skipped 2 previous similar messages [ 279.697064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 279.700015] Lustre: Skipped 1 previous similar message [ 281.544125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 281.548434] Lustre: Skipped 1 previous similar message [ 281.550848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 281.637855] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 286.088481] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 287.818455] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 287.916584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 289.757073] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 289.849379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 291.811061] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 291.812233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.812855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 291.812860] Lustre: Skipped 3 previous similar messages [ 291.822567] Lustre: Skipped 1 previous similar message [ 296.930088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 296.933303] Lustre: Skipped 2 previous similar messages [ 305.119404] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 305.207148] Lustre: server umount lustre-OST0000 complete [ 305.208891] Lustre: Skipped 1 previous similar message [ 307.168854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.169706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.173484] Lustre: Skipped 1 previous similar message [ 307.177135] Lustre: Skipped 3 previous similar messages [ 312.896783] Lustre: server umount lustre-MDT0000 complete [ 314.260892] LustreError: 14312:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322081 with bad export cookie 17165493054421828663 [ 314.263041] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 314.265168] LustreError: 14312:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 315.999043] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:08:03 (1761322083) [ 316.947994] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 317.552847] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:08:04 (1761322084) [ 319.929114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 320.101120] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.122267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 321.702196] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 325.025103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 326.871642] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 329.857331] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 332.016739] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 334.616805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 336.485287] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 339.607372] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 351.200419] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 351.201381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.205337] LustreError: Skipped 1 previous similar message [ 351.210578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 351.214337] Lustre: Skipped 3 previous similar messages [ 352.580469] Lustre: server umount lustre-OST0000 complete [ 352.583649] Lustre: Skipped 1 previous similar message [ 355.807823] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 361.441064] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 361.446341] LustreError: Skipped 1 previous similar message [ 362.174228] LustreError: 18845:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322129 with bad export cookie 17165493054421829517 [ 362.176873] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.178861] LustreError: 18845:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 368.020225] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 369.211290] Key type lgssc unregistered [ 369.335529] LNet: 22520:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.339159] LNet: Removed LNI 192.168.204.123@tcp [ 369.692667] Key type .llcrypt unregistered [ 369.694184] Key type ._llcrypt unregistered [ 380.571594] alg: No test for adler32 (adler32-zlib) [ 381.323422] Key type ._llcrypt registered [ 381.324803] Key type .llcrypt registered [ 381.390913] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 381.786648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 382.053622] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 382.163819] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 382.166372] LNet: Accept secure, port 988 [ 383.759140] Key type lgssc registered [ 384.158815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 387.609924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.794133] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.817614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.564919] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 394.092227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.207902] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 394.249672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 395.916379] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 398.997546] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 401.198322] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 403.669986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 403.812403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.838877] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 405.430377] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 408.496399] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 420.323604] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 420.324109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.331130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 420.333059] Lustre: Skipped 1 previous similar message [ 423.590108] Lustre: server umount lustre-OST0000 complete [ 425.440163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.443656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 425.446074] Lustre: Skipped 2 previous similar messages [ 425.450800] Lustre: Skipped 1 previous similar message [ 430.560491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 430.569927] Lustre: Skipped 1 previous similar message [ 431.267434] Lustre: server umount lustre-MDT0000 complete [ 432.971272] LustreError: 23962:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322199 with bad export cookie 7653166156745691851 [ 432.973771] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 432.974760] LustreError: 23962:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 433.130108] Lustre: server umount lustre-MDT0001 complete [ 438.508732] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 439.669244] Key type lgssc unregistered [ 439.798470] LNet: 27635:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.804481] LNet: Removed LNI 192.168.204.123@tcp [ 440.158709] Key type .llcrypt unregistered [ 440.160687] Key type ._llcrypt unregistered [ 450.227763] alg: No test for adler32 (adler32-zlib) [ 450.979440] Key type ._llcrypt registered [ 450.981076] Key type .llcrypt registered [ 451.026665] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 451.470415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 451.662580] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 451.748709] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 451.751676] LNet: Accept secure, port 988 [ 453.344690] Key type lgssc registered [ 453.817992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 457.400811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.592906] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 458.619432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 460.382945] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 463.840254] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.148270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 464.305218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.082657] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 469.214930] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 471.594982] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 474.476406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 474.620843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.341245] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 479.272379] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 488.996919] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 494.048272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.049711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 494.053125] Lustre: Skipped 2 previous similar messages [ 494.055736] Lustre: Skipped 1 previous similar message [ 494.955688] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 498.441063] Lustre: server umount lustre-MDT0000 complete [ 499.167937] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.174431] LustreError: Skipped 2 previous similar messages [ 500.006186] LustreError: 29077:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322267 with bad export cookie 860017224067835295 [ 500.008167] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.010794] LustreError: 29077:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 500.074498] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 504.289236] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.289501] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.293033] LustreError: Skipped 1 previous similar message [ 504.300360] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 506.200775] Lustre: server umount lustre-MDT0001 complete [ 524.395115] Lustre: server umount lustre-OST0000 complete [ 531.665814] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 532.791311] Key type lgssc unregistered [ 532.914465] LNet: 33094:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.917632] LNet: Removed LNI 192.168.204.123@tcp [ 533.236341] Key type .llcrypt unregistered [ 533.238179] Key type ._llcrypt unregistered [ 543.288997] alg: No test for adler32 (adler32-zlib) [ 544.041437] Key type ._llcrypt registered [ 544.042538] Key type .llcrypt registered [ 544.091451] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 544.491652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 544.727551] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 544.820419] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 544.822366] LNet: Accept secure, port 988 [ 546.407136] Key type lgssc registered [ 546.844743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 550.130291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.333942] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.361400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.070301] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 556.512623] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.707049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.873209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 558.645978] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 561.865502] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 564.114422] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 566.690257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 566.817304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.430850] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 571.270834] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 581.156719] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 586.209536] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 586.210644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.211779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 586.221121] Lustre: Skipped 1 previous similar message [ 587.117349] Lustre: lustre-OST0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 587.121108] Lustre: Skipped 1 previous similar message [ 590.504717] Lustre: server umount lustre-OST0000 complete [ 607.713376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.714940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 607.717776] Lustre: Skipped 1 previous similar message [ 610.250352] Lustre: server umount lustre-MDT0000 complete [ 611.801382] LustreError: 34535:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322378 with bad export cookie 5337090265085655453 [ 611.804255] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.806515] LustreError: 34535:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 626.143131] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 626.251824] Lustre: server umount lustre-MDT0001 complete [ 631.220264] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 632.338192] Key type lgssc unregistered [ 632.469772] LNet: 38401:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.472853] LNet: Removed LNI 192.168.204.123@tcp [ 632.774847] Key type .llcrypt unregistered [ 632.776183] Key type ._llcrypt unregistered [ 642.170928] alg: No test for adler32 (adler32-zlib) [ 642.923212] Key type ._llcrypt registered [ 642.924625] Key type .llcrypt registered [ 642.966368] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 643.295490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 643.541702] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 643.627821] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 643.630472] LNet: Accept secure, port 988 [ 645.215183] Key type lgssc registered [ 645.606648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 648.585712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.759377] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 649.776675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.441875] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 654.816903] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.920120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.083819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 656.854678] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 659.815393] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 662.167229] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 664.880300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 665.001061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.680163] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 669.763743] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 679.335057] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 682.570781] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 683.935181] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 692.197251] LNetError: 120-3: Refusing connection from 192.168.204.123 for 127.0.0.2@tcp: No matching NI [ 692.200984] LNetError: 39166:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 692.204490] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 694.785685] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-2412h14m21s [ 695.407157] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 696.288832] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 696.291956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.296747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 699.873348] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.873537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 699.875797] LNetError: 120-3: Refusing connection from 192.168.204.123 for 127.0.0.2@tcp: No matching NI [ 699.875862] LNetError: 39163:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 699.875884] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 699.879179] Lustre: Skipped 1 previous similar message [ 699.881664] Lustre: Skipped 1 previous similar message [ 702.415339] Lustre: server umount lustre-MDT0000 complete [ 703.825348] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 704.992369] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.994776] LNetError: 120-3: Refusing connection from 192.168.204.123 for 127.0.0.2@tcp: No matching NI [ 704.998267] LustreError: Skipped 2 previous similar messages [ 705.001458] LNetError: 39164:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 705.007139] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 706.378971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 706.450621] LustreError: 11-0: MGC192.168.204.123@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 706.451400] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 706.461146] Lustre: Evicted from MGS (at 192.168.204.123@tcp) after server handle changed from 0xab7f18b7c0d81bc8 to 0xab7f18b7c0d8201a [ 706.466546] Lustre: MGC192.168.204.123@tcp: Connection restored to 192.168.204.123@tcp (at 0@lo) [ 706.566255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 706.609398] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 708.279086] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 709.627735] Lustre: DEBUG MARKER: Wait for df (28828) ... [ 710.254217] Lustre: DEBUG MARKER: done [ 710.638476] LustreError: 41816:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.23@tcp arrived at 1761322477 with bad export cookie 12357623080021401171 [ 710.644274] LustreError: 41816:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 711.650510] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 711.650513] LNetError: 120-3: Refusing connection from 192.168.204.123 for 127.0.0.2@tcp: No matching NI [ 711.650637] LNetError: 39165:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 711.651721] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 711.653464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 711.653980] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 711.655834] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 711.658761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 711.677054] Lustre: Skipped 2 previous similar messages [ 716.256374] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 716.261179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.267583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 717.553705] Lustre: server umount lustre-OST0000 complete [ 721.888933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 721.889294] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 721.889901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 721.889908] Lustre: Skipped 1 previous similar message [ 721.894087] Lustre: Skipped 1 previous similar message [ 721.904465] LNetError: 120-3: Refusing connection from 192.168.204.123 for 127.0.0.2@tcp: No matching NI [ 721.907689] LNetError: 39166:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 721.911600] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 725.329928] Lustre: server umount lustre-MDT0000 complete [ 726.858174] LustreError: 41816:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322493 with bad export cookie 12357623080021401626 [ 726.861224] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 726.863225] LustreError: 41816:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 732.262526] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 733.412373] Key type lgssc unregistered [ 733.542975] LNet: 44974:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.547107] LNet: Removed LNI 192.168.204.123@tcp [ 733.889635] Key type .llcrypt unregistered [ 733.891130] Key type ._llcrypt unregistered [ 747.042106] alg: No test for adler32 (adler32-zlib) [ 747.794396] Key type ._llcrypt registered [ 747.796032] Key type .llcrypt registered [ 747.843152] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 754.014563] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 754.355553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 754.601988] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 754.678255] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 754.680219] LNet: Accept secure, port 988 [ 756.263126] Key type lgssc registered [ 756.622996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 759.787750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 762.268118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 764.554746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 766.931496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 769.408378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 769.436541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.544689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 770.556773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 770.597459] Lustre: lustre-MDT0000: new disk, initializing [ 770.627228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 770.637196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 772.219395] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 776.859435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 776.885090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 776.919800] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 776.940839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 776.943612] Lustre: Skipped 1 previous similar message [ 776.983507] Lustre: lustre-MDT0001: new disk, initializing [ 777.018600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 777.032763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 777.037571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 778.738861] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 782.767229] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 784.823198] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 787.261048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 787.287440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 787.290984] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 787.380645] Lustre: lustre-OST0000: new disk, initializing [ 787.382807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 787.407200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 788.965372] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 792.056170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 792.060995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 792.983709] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 794.510528] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 794.595528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 796.135284] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 796.228534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 797.152744] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 797.153805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 797.155449] LustreError: Skipped 1 previous similar message [ 797.160771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 797.164036] Lustre: Skipped 1 previous similar message [ 799.007032] Lustre: server umount lustre-OST0000 complete [ 802.272233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.272812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 802.276174] Lustre: Skipped 2 previous similar messages [ 806.532655] Lustre: server umount lustre-MDT0000 complete [ 807.392066] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.398136] LustreError: Skipped 1 previous similar message [ 807.904386] LustreError: 49056:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322574 with bad export cookie 18291926101357525785 [ 807.906712] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 807.908765] LustreError: 49056:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 808.040844] Lustre: server umount lustre-MDT0001 complete [ 810.989339] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 12:16:18 (1761322578) [ 813.425447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 813.585260] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.608029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 815.154463] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 818.338274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.474725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 820.014945] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 822.714650] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 824.649481] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 827.043910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 827.161948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.645635] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 831.206383] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 838.399103] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-2412h16m45s [ 838.960059] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 841.564250] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 842.508559] Lustre: DEBUG MARKER: Creating a test file and stat it [ 842.526550] LustreError: 53602:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 847.844447] LNetError: 120-3: Refusing connection from 192.168.204.123 for 127.0.0.2@tcp: No matching NI [ 847.847629] LNetError: 47279:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 847.851144] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 862.575691] Lustre: lustre-MDT0000: Client d9abb6c5-3c5b-4ac8-b3be-c2e3d545b16e (at 192.168.204.23@tcp) reconnecting [ 882.567108] LustreError: 53602:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 883.374574] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 884.284731] Lustre: DEBUG MARKER: done [ 887.777271] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 887.779756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.784505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 887.786713] Lustre: Skipped 1 previous similar message [ 889.824129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.826201] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.826383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.061156] Lustre: server umount lustre-OST0000 complete [ 894.944264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 894.949162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 894.950157] LNetError: 120-3: Refusing connection from 192.168.204.123 for 127.0.0.2@tcp: No matching NI [ 894.950806] Lustre: Skipped 2 previous similar messages [ 894.952736] LNetError: 47276:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 894.955417] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 899.395146] Lustre: server umount lustre-MDT0000 complete [ 900.063980] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.065238] LNetError: 120-3: Refusing connection from 192.168.204.123 for 127.0.0.2@tcp: No matching NI [ 900.068617] LustreError: Skipped 1 previous similar message [ 900.071160] LNetError: 47277:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 900.075490] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 900.683968] LustreError: 53590:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322667 with bad export cookie 18291926101357526639 [ 900.685827] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 900.688932] LustreError: 53590:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 905.785029] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 906.924364] Key type lgssc unregistered [ 907.048496] LNet: 58016:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.052126] LNet: Removed LNI 192.168.204.123@tcp [ 907.379379] Key type .llcrypt unregistered [ 907.380528] Key type ._llcrypt unregistered [ 918.889341] alg: No test for adler32 (adler32-zlib) [ 919.641363] Key type ._llcrypt registered [ 919.642464] Key type .llcrypt registered [ 919.676294] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 925.810021] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 926.342061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.527336] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 926.611789] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 926.613979] LNet: Accept secure, port 988 [ 928.191137] Key type lgssc registered [ 928.630945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 931.881119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 934.368351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 936.565625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 938.842579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 941.258205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 941.285518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 942.385856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 942.396745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 942.436064] Lustre: lustre-MDT0000: new disk, initializing [ 942.460121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.467192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 943.945137] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 948.165556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 948.191694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 948.228517] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 948.246226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 948.248312] Lustre: Skipped 1 previous similar message [ 948.282531] Lustre: lustre-MDT0001: new disk, initializing [ 948.310744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 948.323099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 948.327483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 949.822736] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 953.814864] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 955.704601] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 958.036707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 958.060452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 958.062945] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 958.152612] Lustre: lustre-OST0000: new disk, initializing [ 958.154672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 958.182235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.595155] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 960.629872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 960.632175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 963.178194] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 964.548467] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 964.621386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 965.952143] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 966.030297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 971.234847] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 971.234924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.235874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 971.237714] LustreError: Skipped 1 previous similar message [ 971.241320] Lustre: Skipped 1 previous similar message [ 972.773612] Lustre: server umount lustre-OST0000 complete [ 976.353322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.353879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 976.357279] Lustre: Skipped 1 previous similar message [ 976.362094] Lustre: Skipped 2 previous similar messages [ 980.160681] Lustre: server umount lustre-MDT0000 complete [ 981.381019] LustreError: 62960:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322748 with bad export cookie 15413456183952381311 [ 981.383269] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 981.385181] LustreError: 62960:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 981.532649] Lustre: server umount lustre-MDT0001 complete [ 984.200592] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 12:19:11 (1761322751) [ 984.721573] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 985.309367] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 12:19:12 (1761322752) [ 989.235243] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 990.770917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.934307] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.953071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.313100] Lustre: Failing over lustre-MDT0000 [ 991.314881] LustreError: 67120:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 991.318514] LustreError: 67120:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 991.321422] LustreError: 67120:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 997.855143] Lustre: 67171:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322758/real 1761322758] req@00000000d3b074e2 x1846880704152128/t0(0) o251->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322764 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 997.910341] Lustre: server umount lustre-MDT0000 complete [ 1000.128848] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:19:27 (1761322767) [ 1002.428582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.566454] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.582634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.027101] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1006.821969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.331157] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1010.840149] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1012.652557] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1014.847148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1014.953860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.956797] Lustre: Skipped 1 previous similar message [ 1016.360546] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1018.849767] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1025.960931] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1032.677973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.678237] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1032.679717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1032.684716] Lustre: Skipped 2 previous similar messages [ 1033.934873] Lustre: server umount lustre-MDT0000 complete [ 1035.323260] LustreError: 67721:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322802 with bad export cookie 15413456183952382424 [ 1035.326579] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1035.327548] LustreError: 67721:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1036.648487] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1038.491311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1043.894579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.831214] Lustre: 60330:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322804/real 1761322804] req@00000000ac53f9bc x1846880704166720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322811 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1044.860147] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.007245] Lustre: 60331:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322807/real 1761322807] req@000000004daf3d41 x1846880704166784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322814 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1049.055182] Lustre: 60332:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322809/real 1761322809] req@00000000e04e6131 x1846880704166976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322816 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1049.069306] Lustre: 60332:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1052.068099] Lustre: Evicted from MGS (at 192.168.204.123@tcp) after server handle changed from 0xd5e798bc3d8799d8 to 0xd5e798bc3d87a164 [ 1052.082995] Lustre: MGC192.168.204.123@tcp: Connection restored to 192.168.204.123@tcp (at 0@lo) [ 1052.452044] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.557671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1055.584611] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1055.598470] LustreError: Skipped 1 previous similar message [ 1056.522383] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1060.839664] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.779634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.177226] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1065.131251] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1068.152204] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1075.288705] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1080.792107] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1089.516379] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1089.519511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.532363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1089.535108] Lustre: Skipped 2 previous similar messages [ 1096.160492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.163120] Lustre: Skipped 4 previous similar messages [ 1102.303253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1102.386116] LustreError: 73280:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000811af5a7) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1102.431481] Lustre: server umount lustre-MDT0000 complete [ 1102.437038] Lustre: Skipped 1 previous similar message [ 1105.321443] LustreError: 67719:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322872 with bad export cookie 15413456183952384356 [ 1105.326518] LustreError: 67719:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1105.326563] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1109.516771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1113.503219] Lustre: 60333:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322873/real 1761322873] req@00000000c5a0b1ad x1846880704182784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322880 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1113.517099] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.525567] Lustre: Skipped 2 previous similar messages [ 1115.202516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1122.724886] Lustre: Evicted from MGS (at 192.168.204.123@tcp) after server handle changed from 0xd5e798bc3d87a164 to 0xd5e798bc3d87a76f [ 1122.737542] Lustre: MGC192.168.204.123@tcp: Connection restored to (at 0@lo) [ 1122.740355] Lustre: Skipped 1 previous similar message [ 1122.907992] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1122.917222] LustreError: Skipped 2 previous similar messages [ 1122.957344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1122.960482] Lustre: Skipped 1 previous similar message [ 1123.004340] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1125.288966] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1127.013169] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1128.844157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.063602] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1131.398403] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1135.448835] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1138.047885] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1144.290988] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1144.295802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.305625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.308372] Lustre: Skipped 3 previous similar messages [ 1147.226967] Lustre: server umount lustre-MDT0000 complete [ 1147.228988] Lustre: Skipped 1 previous similar message [ 1149.099322] LustreError: 67720:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322916 with bad export cookie 15413456183952385903 [ 1149.101824] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1149.107752] LustreError: 67720:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1152.740085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1154.706290] Lustre: DEBUG MARKER: files compared the same [ 1161.183198] Lustre: 76861:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322922/real 1761322922] req@000000006c1dfbce x1846880704194304/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322928 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1161.208662] Lustre: 76861:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1172.105566] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1173.494559] Key type lgssc unregistered [ 1173.716760] LNet: 77774:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.723523] LNet: Removed LNI 192.168.204.123@tcp [ 1174.127519] Key type .llcrypt unregistered [ 1174.129180] Key type ._llcrypt unregistered [ 1185.418564] alg: No test for adler32 (adler32-zlib) [ 1186.170507] Key type ._llcrypt registered [ 1186.171916] Key type .llcrypt registered [ 1186.229490] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1186.627662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1186.903588] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1186.998505] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1187.001791] LNet: Accept secure, port 988 [ 1188.607139] Key type lgssc registered [ 1189.085181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1193.036337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.287925] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1194.328230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.461828] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug +malloc all 8 [ 1199.583977] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.310053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.415460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1202.064303] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug +malloc all 8 [ 1205.366967] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1208.140357] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1212.084972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1212.277769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1214.855198] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug +malloc all 8 [ 1215.332077] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1219.801497] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1224.552086] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1229.793218] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1229.798463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.801455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1229.811591] Lustre: Skipped 1 previous similar message [ 1231.180222] Lustre: server umount lustre-OST0000 complete [ 1234.912644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1234.913992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.920875] Lustre: Skipped 1 previous similar message [ 1234.923719] Lustre: Skipped 1 previous similar message [ 1239.709966] Lustre: server umount lustre-MDT0000 complete [ 1240.031923] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.040596] LustreError: Skipped 1 previous similar message [ 1241.718289] LustreError: 79214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323008 with bad export cookie 9018274521806823501 [ 1241.720625] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.724767] LustreError: 79214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1241.888130] Lustre: server umount lustre-MDT0001 complete [ 1249.305247] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1251.144824] Key type lgssc unregistered [ 1251.303711] LNet: 82881:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.310904] LNet: Removed LNI 192.168.204.123@tcp [ 1251.809692] Key type .llcrypt unregistered [ 1251.811298] Key type ._llcrypt unregistered [ 1266.043945] alg: No test for adler32 (adler32-zlib) [ 1266.800860] Key type ._llcrypt registered [ 1266.803100] Key type .llcrypt registered [ 1266.861975] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1267.375185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1267.619992] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1267.743716] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1267.747741] LNet: Accept secure, port 988 [ 1269.367231] Key type lgssc registered [ 1270.053196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1274.871246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1292.447940] LustreError: 84316:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.123@tcp: failed processing log, type 1: rc = -5 [ 1323.168777] LustreError: 84316:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.123@tcp: failed processing log, type 4: rc = -110 [ 1353.887190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.657803] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1358.998284] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1362.789742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.854885] Lustre: *** cfs_fail_loc=706, val=0*** [ 1374.461147] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.503913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1374.554072] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1376.675344] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1379.809411] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.174578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.355108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1381.391634] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1383.617287] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1387.731495] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1390.786390] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1395.168087] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1395.171920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1395.179068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1396.708475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1396.709098] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1396.709463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.711028] Lustre: Skipped 1 previous similar message [ 1397.942339] Lustre: server umount lustre-OST0000 complete [ 1401.824400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.824909] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1401.825217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.828398] Lustre: Skipped 1 previous similar message [ 1406.946188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1406.948767] Lustre: Skipped 2 previous similar messages [ 1412.064265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1412.068270] Lustre: Skipped 1 previous similar message [ 1414.111811] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1414.176460] Lustre: server umount lustre-MDT0000 complete [ 1415.916212] LustreError: 84324:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323182 with bad export cookie 7225216404111224235 [ 1415.920365] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1415.923588] LustreError: 84324:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1416.085767] Lustre: server umount lustre-MDT0001 complete [ 1420.730152] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1421.934350] Key type lgssc unregistered [ 1422.066138] LNet: 88056:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.072335] LNet: Removed LNI 192.168.204.123@tcp [ 1422.416379] Key type .llcrypt unregistered [ 1422.418110] Key type ._llcrypt unregistered [ 1433.821838] alg: No test for adler32 (adler32-zlib) [ 1434.573441] Key type ._llcrypt registered [ 1434.574980] Key type .llcrypt registered [ 1434.625782] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1435.101743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1435.268131] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1435.368828] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1435.371801] LNet: Accept secure, port 988 [ 1436.967143] Key type lgssc registered [ 1437.458451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1441.463468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.554633] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1448.442515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.647330] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.701571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1451.001519] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1454.048683] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.056033] LustreError: Skipped 2 previous similar messages [ 1455.775504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1455.918438] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.925149] LustreError: Skipped 1 previous similar message [ 1455.932195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.121681] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1461.218055] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.226697] LustreError: Skipped 2 previous similar messages [ 1462.614101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.658345] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 1464.683931] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 1464.865527] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1475.042852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.043997] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1475.051800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1475.056475] Lustre: Skipped 1 previous similar message [ 1480.075574] Lustre: server umount lustre-OST0000 complete [ 1485.280973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1485.282712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1485.293051] Lustre: Skipped 2 previous similar messages [ 1485.307276] Lustre: Skipped 1 previous similar message [ 1488.220856] Lustre: server umount lustre-MDT0000 complete [ 1490.629367] Lustre: server umount lustre-MDT0001 complete [ 1498.079536] Lustre: 92918:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323259/real 1761323259] req@00000000616fc4e2 x1846881237872256/t0(0) o251->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323265 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1498.111679] Lustre: server umount MGS complete [ 1503.638916] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:27:50 (1761323270) [ 1518.834152] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 1522.764385] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1527.269828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1529.945769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1532.691735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1535.777557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1538.803604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1538.849458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1540.852172] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1546.571494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1546.612515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1546.732451] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1546.757891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1546.811502] Lustre: lustre-MDT0001: new disk, initializing [ 1546.858253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1546.882893] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1546.888814] LustreError: 97614:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 1549.092359] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1554.964956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1555.007196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1555.011052] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1555.119060] Lustre: lustre-OST0000: new disk, initializing [ 1555.121848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1555.160056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1557.242800] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1562.796965] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1565.221056] Lustre: 99218:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@0000000068fdc542 x1846881237876928/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761323344 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1566.070861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1566.095450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1566.146840] Lustre: lustre-MDT0000: new disk, initializing [ 1566.204616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.216872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1567.264263] Lustre: 99467:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000345e2268 x1846881237879936/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1761323346 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1568.181177] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1572.330054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1572.340061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1574.308540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1574.316666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1594.850234] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1594.853397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1594.865866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1596.896078] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1596.902140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.908055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1599.799660] Lustre: server umount lustre-OST0000 complete [ 1602.528716] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1602.528929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1602.540168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1602.544755] Lustre: Skipped 1 previous similar message [ 1602.547464] Lustre: Skipped 2 previous similar messages [ 1624.032607] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1624.142614] Lustre: server umount lustre-MDT0001 complete [ 1624.144159] Lustre: Skipped 1 previous similar message [ 1631.199180] Lustre: 100661:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323392/real 1761323392] req@0000000030cc4e5e x1846881237889664/t0(0) o251->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323398 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1636.234665] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 12:30:03 (1761323403) [ 1644.705463] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1645.801821] Key type lgssc unregistered [ 1645.919395] LNet: 102066:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.922986] LNet: Removed LNI 192.168.204.123@tcp [ 1646.245115] Key type .llcrypt unregistered [ 1646.246805] Key type ._llcrypt unregistered [ 1653.966427] alg: No test for adler32 (adler32-zlib) [ 1654.718464] Key type ._llcrypt registered [ 1654.719519] Key type .llcrypt registered [ 1654.766231] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1655.156330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1655.389276] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1655.485925] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1655.487802] LNet: Accept secure, port 988 [ 1657.087266] Key type lgssc registered [ 1657.535511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1660.975427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.186625] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1662.206628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1663.961256] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1667.553114] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1667.766259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.938383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1669.774316] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1673.065316] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1675.661335] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1678.567642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1678.707984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1680.636577] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1683.886063] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1692.011340] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 1699.298390] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1699.301965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.308161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1700.322411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1700.324528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.790514] Lustre: server umount lustre-OST0000 complete [ 1709.025263] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1709.028629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.035243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.037704] Lustre: Skipped 1 previous similar message [ 1715.680250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1715.682530] Lustre: Skipped 3 previous similar messages [ 1720.287184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1720.343452] Lustre: server umount lustre-MDT0000 complete [ 1720.800671] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1720.806581] LustreError: Skipped 1 previous similar message [ 1721.858213] LustreError: 103313:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323488 with bad export cookie 16399014377376860591 [ 1721.860871] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1721.863403] LustreError: 103313:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1722.011868] Lustre: server umount lustre-MDT0001 complete [ 1727.409532] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1728.527535] Key type lgssc unregistered [ 1728.648422] LNet: 107006:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.651497] LNet: Removed LNI 192.168.204.123@tcp [ 1728.972958] Key type .llcrypt unregistered [ 1728.974580] Key type ._llcrypt unregistered [ 1736.165662] alg: No test for adler32 (adler32-zlib) [ 1736.917432] Key type ._llcrypt registered [ 1736.919205] Key type .llcrypt registered [ 1736.964635] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1737.462156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1737.618036] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1737.704345] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1737.706882] LNet: Accept secure, port 988 [ 1739.295184] Key type lgssc registered [ 1739.740890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1742.488131] LustreError: 108135:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1742.492439] LustreError: 108136:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1742.496333] LustreError: 108135:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4995 [ 1742.501731] LustreError: 108135:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1742.506811] LustreError: 108135:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1742.511360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.733051] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.757942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.811464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.940226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1748.681545] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1750.858804] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing lsmod [ 1752.016789] LustreError: 109440:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1752.024545] LustreError: 109444:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1752.027033] LustreError: 109440:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4993 [ 1752.030499] LustreError: 109440:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1752.036980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1752.037039] LustreError: 109440:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1752.177753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.151934] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1757.155501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.161377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.664909] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.666543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.669505] Lustre: Skipped 1 previous similar message [ 1759.366675] Lustre: server umount lustre-MDT0000 complete [ 1760.759395] LustreError: 109483:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323527 with bad export cookie 1152426264990463144 [ 1760.760665] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1760.764664] LustreError: 109483:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1760.894600] Lustre: server umount lustre-MDT0001 complete [ 1768.927183] Lustre: 110033:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323529/real 1761323529] req@00000000f9003478 x1846881554538048/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761323535 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1769.009988] Lustre: server umount lustre-OST0000 complete [ 1772.319258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.507570] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1772.539049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.166138] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1777.632970] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1777.766933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1777.940076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1779.771931] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1783.028801] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1785.608162] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1788.676450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1788.825994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.566869] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1793.765058] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1802.156387] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 1807.328359] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1807.329944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.335167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1807.339625] Lustre: Skipped 1 previous similar message [ 1807.342732] Lustre: Skipped 1 previous similar message [ 1808.113816] Lustre: server umount lustre-OST0000 complete [ 1811.345413] LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 [ 1811.345642] LustreError: 112456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323578 with bad export cookie 1152426264990463816 [ 1811.352940] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1811.353896] LustreError: 112456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1816.700616] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1817.950729] Key type lgssc unregistered [ 1818.083889] LNet: 114178:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.087768] LNet: Removed LNI 192.168.204.123@tcp [ 1818.462986] Key type .llcrypt unregistered [ 1818.464603] Key type ._llcrypt unregistered [ 1828.760524] alg: No test for adler32 (adler32-zlib) [ 1829.512428] Key type ._llcrypt registered [ 1829.515524] Key type .llcrypt registered [ 1829.564920] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1829.946454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1830.203187] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1830.308762] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1830.312020] LNet: Accept secure, port 988 [ 1831.912130] Key type lgssc registered [ 1832.406666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1836.087295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.298670] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.332979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1839.318255] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1842.656191] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1843.038509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.212270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1844.924962] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1848.153148] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1850.700405] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1853.826126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1853.997451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.962898] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1859.232938] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1867.302819] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1871.945118] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1872.949668] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1874.915897] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1874.919650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.930677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.472283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.472765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.300903] Lustre: server umount lustre-OST0000 complete [ 1882.592514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.593317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.597629] Lustre: Skipped 1 previous similar message [ 1882.599549] Lustre: Skipped 1 previous similar message [ 1887.712650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1887.716148] Lustre: Skipped 2 previous similar messages [ 1887.943751] Lustre: server umount lustre-MDT0000 complete [ 1889.350015] LustreError: 118273:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323656 with bad export cookie 18293637444733590561 [ 1889.352800] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1889.355475] LustreError: 118273:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1889.488510] Lustre: server umount lustre-MDT0001 complete [ 1894.870851] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1896.086093] Key type lgssc unregistered [ 1896.211448] LNet: 119430:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.215055] LNet: Removed LNI 192.168.204.123@tcp [ 1896.535791] Key type .llcrypt unregistered [ 1896.537284] Key type ._llcrypt unregistered [ 1904.728266] alg: No test for adler32 (adler32-zlib) [ 1905.480377] Key type ._llcrypt registered [ 1905.482014] Key type .llcrypt registered [ 1905.537160] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1905.995685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1906.187766] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1906.277587] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1906.280218] LNet: Accept secure, port 988 [ 1907.887134] Key type lgssc registered [ 1908.400062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1911.785116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.976103] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1912.996611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1914.765793] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1918.433212] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.462629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.637110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1920.421928] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1923.644221] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1925.979144] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1928.873249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1929.019868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.725982] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1933.691569] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1943.399132] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1948.641392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.642470] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1948.642532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1948.646696] Lustre: Skipped 1 previous similar message [ 1949.347093] Lustre: server umount lustre-OST0000 complete [ 1953.761080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.761698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.765908] Lustre: Skipped 1 previous similar message [ 1953.768448] Lustre: Skipped 1 previous similar message [ 1956.961607] Lustre: server umount lustre-MDT0000 complete [ 1958.516065] LustreError: 120677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323725 with bad export cookie 1793127849935869361 [ 1958.518753] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1958.520502] LustreError: 120677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1958.661444] Lustre: server umount lustre-MDT0001 complete [ 1964.066645] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 1965.250318] Key type lgssc unregistered [ 1965.391944] LNet: 124367:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.396739] LNet: Removed LNI 192.168.204.123@tcp [ 1965.750571] Key type .llcrypt unregistered [ 1965.752733] Key type ._llcrypt unregistered [ 1976.143512] alg: No test for adler32 (adler32-zlib) [ 1976.895371] Key type ._llcrypt registered [ 1976.896798] Key type .llcrypt registered [ 1976.939431] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 1977.374878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1977.544044] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1977.632144] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 1977.634718] LNet: Accept secure, port 988 [ 1979.223127] Key type lgssc registered [ 1979.624105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1982.757657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.948868] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.972029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.593502] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1988.841141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.998648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1990.796105] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 1994.358463] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1997.084693] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2000.106529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.268065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.288837] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2003.367564] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 2005.675539] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2009.504143] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2010.308581] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 2011.053826] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2011.056989] Lustre: Skipped 2 previous similar messages [ 2012.778667] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2012.781574] Lustre: Skipped 2 previous similar messages [ 2018.811886] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2018.823143] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2018.825458] Lustre: Skipped 1 previous similar message [ 2022.082878] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2022.085867] Lustre: Skipped 2 previous similar messages [ 2027.522988] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.23@tcp 192.168.0.[2,10]@tcp [ 2031.396784] Lustre: server umount lustre-OST0000 complete [ 2034.145620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.147266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2034.150797] Lustre: Skipped 1 previous similar message [ 2038.239158] Lustre: 129930:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323799/real 1761323799] req@000000003d4f8a46 x1846881806210880/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761323805 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2038.325137] Lustre: server umount lustre-MDT0000 complete [ 2039.264072] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.268779] LustreError: Skipped 1 previous similar message [ 2039.637939] LustreError: 125809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323806 with bad export cookie 12844525214366501209 [ 2039.640324] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.642541] LustreError: 125809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2045.919183] Lustre: 130132:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323806/real 1761323806] req@0000000055874b52 x1846881806212096/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761323812 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2046.061697] Lustre: server umount lustre-MDT0001 complete [ 2051.389791] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2052.486379] Key type lgssc unregistered [ 2052.611334] LNet: 130734:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.614663] LNet: Removed LNI 192.168.204.123@tcp [ 2052.920545] Key type .llcrypt unregistered [ 2052.922151] Key type ._llcrypt unregistered [ 2065.021345] alg: No test for adler32 (adler32-zlib) [ 2065.773450] Key type ._llcrypt registered [ 2065.774935] Key type .llcrypt registered [ 2065.820619] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2066.183803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2066.410337] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2066.499140] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2066.501364] LNet: Accept secure, port 988 [ 2068.087122] Key type lgssc registered [ 2068.506027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2071.474568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.628989] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.640830] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2072.645824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.650531] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.204.23@tcp 192.168.0.[2,10]@tcp [ 2074.119349] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2077.314253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.430765] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2077.441821] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.23@tcp 192.168.0.[2,10]@tcp [ 2077.458583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2079.048181] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2081.941339] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2084.232124] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2086.854572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2086.994817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.042620] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 2088.760319] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2091.583443] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2097.633578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.638613] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2097.638947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2097.644802] Lustre: Skipped 1 previous similar message [ 2101.928603] Lustre: server umount lustre-OST0000 complete [ 2107.872194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.872792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2107.876811] Lustre: Skipped 2 previous similar messages [ 2107.880870] Lustre: Skipped 1 previous similar message [ 2109.381480] Lustre: server umount lustre-MDT0000 complete [ 2110.771885] LustreError: 135244:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323877 with bad export cookie 543098626372817364 [ 2110.774133] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2110.777067] LustreError: 135244:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2110.916649] Lustre: server umount lustre-MDT0001 complete [ 2116.259821] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2117.380172] Key type lgssc unregistered [ 2117.507366] LNet: 136046:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.511690] LNet: Removed LNI 192.168.204.123@tcp [ 2117.836112] Key type .llcrypt unregistered [ 2117.837706] Key type ._llcrypt unregistered [ 2128.396552] alg: No test for adler32 (adler32-zlib) [ 2129.148469] Key type ._llcrypt registered [ 2129.150208] Key type .llcrypt registered [ 2129.200837] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 2134.918291] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2135.370232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2135.591070] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2135.686284] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2135.688689] LNet: Accept secure, port 988 [ 2137.279145] Key type lgssc registered [ 2137.702270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2140.974000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2143.259270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2145.314184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2147.433787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2149.912816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2149.943495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.050146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2151.061898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2151.102441] Lustre: lustre-MDT0000: new disk, initializing [ 2151.131867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.139950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2152.715876] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2157.326209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2157.353443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.386947] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2157.408492] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2157.411129] Lustre: Skipped 1 previous similar message [ 2157.449117] Lustre: lustre-MDT0001: new disk, initializing [ 2157.486234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2157.500046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2157.504779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2159.055481] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2163.152171] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2165.240869] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2167.736670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2167.767270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2167.770775] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2167.872768] Lustre: lustre-OST0000: new disk, initializing [ 2167.875665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2167.906536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.644722] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2173.957446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2173.963494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2174.221694] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2176.150996] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2176.253881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2178.138486] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2178.230243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2179.041240] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2179.041693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.043491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.044558] LustreError: Skipped 1 previous similar message [ 2179.049799] Lustre: Skipped 1 previous similar message [ 2184.162826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2184.165615] Lustre: Skipped 2 previous similar messages [ 2189.282984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.285033] Lustre: Skipped 1 previous similar message [ 2193.375792] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2193.456875] Lustre: server umount lustre-OST0000 complete [ 2198.498153] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2198.501770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.507604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2199.520491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2204.640556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2204.644199] Lustre: Skipped 3 previous similar messages [ 2209.247234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2209.300588] Lustre: server umount lustre-MDT0000 complete [ 2209.760422] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.765860] LustreError: Skipped 1 previous similar message [ 2210.881443] LustreError: 139553:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323977 with bad export cookie 17701083807724330818 [ 2210.885530] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2210.885785] LustreError: 139553:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2211.050761] Lustre: server umount lustre-MDT0001 complete [ 2214.582351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.770787] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.800803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.663478] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2220.000804] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.515381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.712569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2222.531899] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2225.575766] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2227.878473] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2230.694865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2230.818657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2232.580227] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2235.731663] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2237.636610] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2245.964819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2247.717650] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2247.812315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2251.968199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2252.001381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2252.005479] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2252.071628] Lustre: lustre-OST0001: new disk, initializing [ 2252.075090] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2252.117546] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2253.886787] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2257.816688] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2260.485802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2260.490239] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2260.988324] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2262.658570] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 2262.758088] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2264.849051] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff901d06b6b000.ost_server_uuid 40 [ 2265.405394] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff901d06b6b000.ost_server_uuid in FULL state after 0 sec [ 2270.690855] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2270.692711] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2270.693858] LustreError: Skipped 1 previous similar message [ 2270.699930] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2270.703454] Lustre: Skipped 1 previous similar message [ 2273.528441] Lustre: server umount lustre-OST0001 complete [ 2275.809430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.809618] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2275.809864] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.813967] Lustre: Skipped 1 previous similar message [ 2280.929457] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2280.936247] LustreError: Skipped 2 previous similar messages [ 2289.119179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2289.196655] Lustre: server umount lustre-OST0000 complete [ 2291.169545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.170494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2291.175438] Lustre: Skipped 2 previous similar messages [ 2291.177685] Lustre: Skipped 6 previous similar messages [ 2298.195269] LustreError: 145869:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324065 with bad export cookie 17701083807724331658 [ 2298.197226] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2298.199320] LustreError: 145869:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2298.343840] Lustre: server umount lustre-MDT0001 complete [ 2298.345519] Lustre: Skipped 1 previous similar message [ 2306.390299] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2307.408941] Key type lgssc unregistered [ 2307.519363] LNet: 150642:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.522888] LNet: Removed LNI 192.168.204.123@tcp [ 2307.819865] Key type .llcrypt unregistered [ 2307.821229] Key type ._llcrypt unregistered [ 2319.197916] alg: No test for adler32 (adler32-zlib) [ 2319.949475] Key type ._llcrypt registered [ 2319.950915] Key type .llcrypt registered [ 2319.986702] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 2325.879180] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2326.282539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2326.390250] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2326.467456] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2326.469974] LNet: Accept secure, port 988 [ 2328.047128] Key type lgssc registered [ 2328.394204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2331.359861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2333.561490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2335.578493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2337.706283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2340.171045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2340.195583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.302194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2341.313113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2341.350505] Lustre: lustre-MDT0000: new disk, initializing [ 2341.376976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.385327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2343.119413] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2347.586501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2347.607861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2347.641098] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2347.656984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2347.658927] Lustre: Skipped 1 previous similar message [ 2347.699934] Lustre: lustre-MDT0001: new disk, initializing [ 2347.736126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2347.749832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2347.753398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2349.271916] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2353.457127] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2355.716432] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2358.175156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2358.196883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2358.199481] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2358.288633] Lustre: lustre-OST0000: new disk, initializing [ 2358.290824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2358.316870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2359.873494] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2363.836701] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2365.371088] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2365.430755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2365.433714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2366.472745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2367.849865] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2367.933349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2370.530675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.530874] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2370.534547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2370.540030] Lustre: Skipped 1 previous similar message [ 2374.702304] Lustre: server umount lustre-OST0000 complete [ 2378.719734] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2378.722650] LustreError: Skipped 1 previous similar message [ 2378.723813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.727948] Lustre: Skipped 1 previous similar message [ 2378.730428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.768019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.768303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.209930] Lustre: server umount lustre-MDT0000 complete [ 2383.694421] LustreError: 154724:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324150 with bad export cookie 12222733481784400969 [ 2383.698471] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.699181] LustreError: 154724:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2383.850307] Lustre: server umount lustre-MDT0001 complete [ 2386.670540] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 12:42:33 (1761324153) [ 2391.865259] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 2394.662323] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2397.874417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2400.009075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2402.052848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2404.163968] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2406.543492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2406.565647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.645435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2406.656582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2406.694609] Lustre: lustre-MDT0000: new disk, initializing [ 2406.721175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.728437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2408.229327] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2412.484232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2412.513084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.547345] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2412.564296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2412.567142] Lustre: Skipped 1 previous similar message [ 2412.605351] Lustre: lustre-MDT0001: new disk, initializing [ 2412.637113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2412.648481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2412.652196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2414.229793] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2418.565778] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2420.825170] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2423.355257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2423.382486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2423.385323] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2423.478251] Lustre: lustre-OST0000: new disk, initializing [ 2423.480695] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2423.510944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2425.168533] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2429.238805] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2430.455291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2430.458460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2436.812483] Lustre: Failing over lustre-OST0000 [ 2436.892144] Lustre: server umount lustre-OST0000 complete [ 2447.327212] Lustre: 152957:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324207/real 1761324207] req@00000000a148826c x1846882172170048/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761324214 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2447.335188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.351095] Lustre: 152959:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324207/real 1761324207] req@00000000f9f91420 x1846882172170432/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1761324214 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2448.359136] Lustre: 152959:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2450.097030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2450.196648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.203289] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2451.632561] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2452.255075] Lustre: 152956:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324212/real 1761324212] req@000000002c5fca91 x1846882172171008/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761324219 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2452.920617] Lustre: Failing over lustre-MDT0000 [ 2453.079915] Lustre: server umount lustre-MDT0000 complete [ 2453.281626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2453.282530] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.286471] Lustre: Skipped 2 previous similar messages [ 2454.303150] Lustre: 152958:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324214/real 1761324214] req@0000000020e4fa0b x1846882172171456/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761324221 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2454.304177] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.305656] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2454.313262] Lustre: 152958:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2454.317775] LustreError: Skipped 5 previous similar messages [ 2459.489180] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.495463] LustreError: Skipped 1 previous similar message [ 2460.639222] Lustre: 152959:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324220/real 1761324220] req@000000008206e5f7 x1846882172174528/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324227 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2460.647985] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2465.632605] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.638726] LustreError: Skipped 7 previous similar messages [ 2466.523354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.786450] Lustre: Evicted from MGS (at 192.168.204.123@tcp) after server handle changed from 0xa99fdf57c6085f91 to 0xa99fdf57c6086437 [ 2466.792317] Lustre: MGC192.168.204.123@tcp: Connection restored to (at 0@lo) [ 2466.858585] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2466.902200] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2468.378816] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2471.776774] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2471.907138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2471.911787] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2471.914816] Lustre: Skipped 1 previous similar message [ 2471.933611] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2477.024406] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2477.025790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.026927] LustreError: Skipped 1 previous similar message [ 2477.031417] Lustre: Skipped 1 previous similar message [ 2477.032068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2477.037310] Lustre: Skipped 2 previous similar messages [ 2479.064473] Lustre: server umount lustre-OST0000 complete [ 2482.144240] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2482.145191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2482.149272] Lustre: Skipped 1 previous similar message [ 2487.264035] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.267831] LustreError: Skipped 8 previous similar messages [ 2487.625438] LustreError: 161710:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324254 with bad export cookie 12222733481784402999 [ 2487.627889] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2487.629448] LustreError: 161710:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2492.296120] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2493.215589] Key type lgssc unregistered [ 2493.325305] LNet: 167555:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2493.328568] LNet: Removed LNI 192.168.204.123@tcp [ 2493.589293] Key type .llcrypt unregistered [ 2493.590553] Key type ._llcrypt unregistered [ 2501.877392] alg: No test for adler32 (adler32-zlib) [ 2502.629333] Key type ._llcrypt registered [ 2502.630456] Key type .llcrypt registered [ 2502.668022] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2502.960855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2503.173129] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2503.251606] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2503.253657] LNet: Accept secure, port 988 [ 2504.831079] Key type lgssc registered [ 2505.140938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2507.561031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.703439] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.718445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2510.127992] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2512.808449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.913564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2514.231174] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2516.657699] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2518.417510] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2520.620736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2520.713186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.130589] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2523.814824] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2524.590130] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2619.569095] Lustre: Failing over lustre-MDT0000 [ 2619.677906] Lustre: server umount lustre-MDT0000 complete [ 2620.383716] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2620.385612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.388583] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.408243] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.408834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.411522] LustreError: Skipped 1 previous similar message [ 2621.416978] Lustre: Skipped 1 previous similar message [ 2626.528332] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.531658] LustreError: Skipped 4 previous similar messages [ 2628.575106] Lustre: 168329:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324388/real 1761324388] req@00000000c36bebc8 x1846882358050240/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324395 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2628.581686] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2632.655498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.695993] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.698864] LustreError: Skipped 7 previous similar messages [ 2634.721266] Lustre: Evicted from MGS (at 192.168.204.123@tcp) after server handle changed from 0xed562cddfbfaba10 to 0xed562cddfbfd2623 [ 2634.726782] Lustre: MGC192.168.204.123@tcp: Connection restored to 192.168.204.123@tcp (at 0@lo) [ 2634.803882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.834946] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2635.616844] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2636.174206] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2639.848607] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2639.855410] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2639.870371] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 2641.787290] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2642.348274] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2655.201660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.201797] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2655.203577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2655.207602] Lustre: Skipped 1 previous similar message [ 2660.320144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2660.323440] Lustre: Skipped 4 previous similar messages [ 2661.127506] Lustre: server umount lustre-MDT0000 complete [ 2662.352875] LustreError: 168994:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324429 with bad export cookie 17101905966913103395 [ 2662.355382] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.355815] LustreError: 168994:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2665.439873] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.440098] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2665.443365] LustreError: Skipped 3 previous similar messages [ 2665.449809] Lustre: Skipped 1 previous similar message [ 2665.453809] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2667.488107] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2668.541175] Lustre: server umount lustre-MDT0001 complete [ 2684.383137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2684.461329] Lustre: server umount lustre-OST0000 complete [ 2687.712342] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 2690.560392] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2693.750641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2695.884410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2697.859049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2699.908751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2702.122177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2702.145803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.232785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2702.245719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2702.284423] Lustre: lustre-MDT0000: new disk, initializing [ 2702.308625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.316552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2703.735813] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2707.981418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2708.004336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.039306] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2708.056462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2708.058818] Lustre: Skipped 1 previous similar message [ 2708.093384] Lustre: lustre-MDT0001: new disk, initializing [ 2708.130911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2708.133572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2709.627663] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2713.470158] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2715.344730] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2717.600527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2717.627182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2717.629870] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2717.723636] Lustre: lustre-OST0000: new disk, initializing [ 2717.725611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2717.751820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2717.754656] Lustre: Skipped 1 previous similar message [ 2717.815179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2717.818212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2719.220834] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2722.955578] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2724.349795] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2724.421868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2725.878291] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2725.955289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2728.416417] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2728.417928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.418627] LustreError: Skipped 1 previous similar message [ 2728.423864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2728.426294] Lustre: Skipped 1 previous similar message [ 2728.755861] Lustre: server umount lustre-OST0000 complete [ 2738.655732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2738.657508] Lustre: Skipped 3 previous similar messages [ 2744.287115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2744.322754] Lustre: server umount lustre-MDT0000 complete [ 2744.800790] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.806059] LustreError: Skipped 2 previous similar messages [ 2745.530068] LustreError: 176717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324512 with bad export cookie 17101905966913104123 [ 2745.532844] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.534417] LustreError: 176717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2748.332949] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:48:35 (1761324515) [ 2753.497504] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 2756.219927] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2759.343236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2761.652338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2763.766621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2766.113920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2768.735878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2768.763386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.864233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2768.875736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2768.918780] Lustre: lustre-MDT0000: new disk, initializing [ 2768.946918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.954603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2770.602315] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2775.124155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2775.150912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.193186] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2775.211757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2775.214289] Lustre: Skipped 1 previous similar message [ 2775.254729] Lustre: lustre-MDT0001: new disk, initializing [ 2775.306639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2775.311090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2776.856034] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2780.655422] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2782.475377] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2784.703536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2784.723968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2784.727849] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2784.826743] Lustre: lustre-OST0000: new disk, initializing [ 2784.829387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2785.910610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2785.914933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2786.247823] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2790.023138] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2796.001545] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2796.004047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2796.007924] Lustre: Skipped 3 previous similar messages [ 2796.009418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2796.011209] Lustre: Skipped 2 previous similar messages [ 2798.054731] Lustre: server umount lustre-OST0000 complete [ 2798.056554] Lustre: Skipped 1 previous similar message [ 2801.119955] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2801.122013] LustreError: Skipped 1 previous similar message [ 2806.751824] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2806.754793] LustreError: Skipped 1 previous similar message [ 2806.755924] LustreError: 182839:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324573 with bad export cookie 17101905966913104963 [ 2806.758190] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2806.758632] LustreError: 182839:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2809.580595] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:49:36 (1761324576) [ 2815.118372] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 2817.961965] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2821.270844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2823.527203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2825.560563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2827.582858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2829.771154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2829.790722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.865745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2829.874861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2829.912751] Lustre: lustre-MDT0000: new disk, initializing [ 2829.935457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2829.937205] Lustre: Skipped 2 previous similar messages [ 2829.941747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2831.279363] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2835.307158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2835.330397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.441265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2836.807682] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2840.492975] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2842.286310] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2844.447598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2844.471862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2844.475930] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2845.885844] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2849.584599] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2851.825628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2856.929628] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2856.930609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.931995] LustreError: Skipped 1 previous similar message [ 2856.938183] Lustre: Skipped 3 previous similar messages [ 2856.940558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2856.944703] Lustre: Skipped 4 previous similar messages [ 2858.596314] Lustre: server umount lustre-OST0000 complete [ 2858.598278] Lustre: Skipped 2 previous similar messages [ 2861.278936] LustreError: 189293:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324628 with bad export cookie 17101905966913106041 [ 2861.281816] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2861.284338] LustreError: 189293:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2872.174293] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2873.164833] Key type lgssc unregistered [ 2873.269358] LNet: 193256:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2873.271750] LNet: Removed LNI 192.168.204.123@tcp [ 2873.556872] Key type .llcrypt unregistered [ 2873.558447] Key type ._llcrypt unregistered [ 2881.918229] alg: No test for adler32 (adler32-zlib) [ 2882.670327] Key type ._llcrypt registered [ 2882.671165] Key type .llcrypt registered [ 2882.705489] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2883.111405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2883.229516] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2883.296097] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2883.298568] LNet: Accept secure, port 988 [ 2884.879119] Key type lgssc registered [ 2885.193616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2887.779431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.930487] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.946656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.348479] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2893.265371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.387917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2894.768779] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2897.218994] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2898.954517] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2901.100697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2901.186279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.647615] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2904.293834] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2905.306470] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2924.000556] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2924.001810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2924.003736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2924.006392] Lustre: Skipped 1 previous similar message [ 2924.012413] Lustre: Skipped 1 previous similar message [ 2927.907690] Lustre: server umount lustre-OST0000 complete [ 2934.240208] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2934.240500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2934.247858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2934.249801] Lustre: Skipped 1 previous similar message [ 2935.306441] Lustre: server umount lustre-MDT0000 complete [ 2936.614200] LustreError: 195501:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324703 with bad export cookie 16264215935768530621 [ 2936.616728] LustreError: 195501:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2936.616982] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.751521] Lustre: server umount lustre-MDT0001 complete [ 2941.091411] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 2941.983996] Key type lgssc unregistered [ 2942.098334] LNet: 198388:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2942.100764] LNet: Removed LNI 192.168.204.123@tcp [ 2942.372914] Key type .llcrypt unregistered [ 2942.373955] Key type ._llcrypt unregistered [ 2950.572249] alg: No test for adler32 (adler32-zlib) [ 2951.323281] Key type ._llcrypt registered [ 2951.324156] Key type .llcrypt registered [ 2951.357773] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 2951.685526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2951.880187] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2951.950954] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 2951.952361] LNet: Accept secure, port 988 [ 2953.527104] Key type lgssc registered [ 2953.880901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2956.333984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.472170] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.484907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2958.846331] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2961.543964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2961.652073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2962.997517] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2965.287222] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2966.983988] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2969.080404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2969.180517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2970.527938] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 2972.260405] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 2972.924453] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2981.857028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.857615] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2981.860623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2981.863757] Lustre: Skipped 1 previous similar message [ 2984.296063] Lustre: lustre-OST0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 2985.946512] Lustre: server umount lustre-OST0000 complete [ 2988.376614] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff901d21e5f800.ost_server_uuid 40 [ 2988.871921] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff901d21e5f800.ost_server_uuid in DISCONN state after 0 sec [ 2989.356291] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3002.847695] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3002.849859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.854068] Lustre: Skipped 1 previous similar message [ 3002.855789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.569289] Lustre: server umount lustre-MDT0000 complete [ 3006.765137] LustreError: 199827:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324773 with bad export cookie 15959907962913451725 [ 3006.767555] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3006.769310] LustreError: 199827:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3006.892328] Lustre: server umount lustre-MDT0001 complete [ 3009.723258] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:52:57 (1761324777) [ 3011.809093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.981356] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3012.002661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.373284] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3016.102444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3016.218732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3017.565543] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3020.057937] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3021.926345] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3024.152021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3024.259376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3025.727966] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3027.365318] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 3028.274553] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3030.541598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3030.565078] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3030.567812] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3030.622863] Lustre: lustre-OST0001: new disk, initializing [ 3030.625661] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3031.723416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3031.728857] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3032.175899] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3036.057601] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3037.154319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3037.155556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3037.158699] Lustre: Skipped 2 previous similar messages [ 3037.163137] Lustre: Skipped 3 previous similar messages [ 3041.642672] Lustre: lustre-OST0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 3043.062453] Lustre: server umount lustre-OST0000 complete [ 3045.212601] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3045.290134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3046.710107] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3046.753650] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.785609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3051.873224] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3051.877393] LustreError: Skipped 2 previous similar messages [ 3057.632950] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3057.633178] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.633283] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3057.634116] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3057.634121] Lustre: Skipped 2 previous similar messages [ 3057.636628] LustreError: Skipped 1 previous similar message [ 3057.640327] LustreError: Skipped 3 previous similar messages [ 3062.882342] Lustre: server umount lustre-OST0001 complete [ 3067.359899] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3067.363493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.368391] Lustre: Skipped 1 previous similar message [ 3071.473931] LustreError: 203842:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324838 with bad export cookie 15959907962913452775 [ 3071.476183] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3071.478011] LustreError: 203842:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3071.607025] Lustre: server umount lustre-MDT0001 complete [ 3071.608967] Lustre: Skipped 1 previous similar message [ 3073.457579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3074.761377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3076.005447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3077.249996] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3079.360716] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:54:06 (1761324846) [ 3081.683755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.772717] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3081.785219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3081.846525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.849046] Lustre: Skipped 1 previous similar message [ 3083.225684] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3086.085140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.122114] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3086.139674] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3087.706808] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3090.268802] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3092.208960] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3094.322855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3094.380817] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3095.739491] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3098.234974] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3100.488179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3100.508647] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3101.847078] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 3101.856095] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 3101.952835] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3104.382361] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3107.297972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.298592] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3107.302018] Lustre: Skipped 1 previous similar message [ 3107.304516] LustreError: Skipped 1 previous similar message [ 3107.305220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3107.308533] Lustre: Skipped 6 previous similar messages [ 3111.284941] Lustre: server umount lustre-OST0000 complete [ 3112.419739] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3112.424137] LustreError: Skipped 3 previous similar messages [ 3122.656243] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3122.658189] LustreError: Skipped 1 previous similar message [ 3132.383796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3132.389032] Lustre: Skipped 2 previous similar messages [ 3136.648827] LustreError: 210255:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324903 with bad export cookie 15959907962913454140 [ 3136.651762] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3136.652690] LustreError: 210255:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3138.714562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3140.023751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3141.303159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3142.594201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3144.837279] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:55:12 (1761324912) [ 3150.088285] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3152.805383] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3155.779065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3157.861178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3159.774984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3161.677223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3163.813188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3163.840750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.919119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3163.927945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3163.965841] Lustre: lustre-MDT0000: new disk, initializing [ 3163.987875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3163.990252] Lustre: Skipped 3 previous similar messages [ 3163.996325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3165.376568] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3169.475491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3169.497722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.526435] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3169.539881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3169.541552] Lustre: Skipped 1 previous similar message [ 3169.572092] Lustre: lustre-MDT0001: new disk, initializing [ 3169.605698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3169.608584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3170.949256] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3174.573979] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3176.347504] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3178.518208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3178.543534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3178.545965] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3178.631523] Lustre: lustre-OST0000: new disk, initializing [ 3178.633907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3180.083819] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3181.811983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3181.814552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3183.822597] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3185.340187] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3185.406730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3186.797281] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3186.876866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3192.290135] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3192.290686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.292829] LustreError: Skipped 2 previous similar messages [ 3192.295837] Lustre: Skipped 1 previous similar message [ 3192.296502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3192.299465] Lustre: Skipped 8 previous similar messages [ 3193.634814] Lustre: server umount lustre-OST0000 complete [ 3193.636389] Lustre: Skipped 3 previous similar messages [ 3209.183131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3210.459143] LustreError: 219311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324977 with bad export cookie 15959907962913455764 [ 3210.461541] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3210.462892] LustreError: 219311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3213.275249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.408079] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.413954] LustreError: Skipped 8 previous similar messages [ 3214.751953] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3217.491486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.978764] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3221.359212] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3223.120733] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3225.274087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3226.692716] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3229.114741] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3230.502320] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3230.567224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3231.967338] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3233.052532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 3233.759802] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3233.764028] LustreError: Skipped 1 previous similar message [ 3242.004691] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3242.087591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3243.588898] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3267.270599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3267.378543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.380605] Lustre: Skipped 5 previous similar messages [ 3268.914128] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3270.434496] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3270.438571] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 3270.440483] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3271.408776] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3276.017513] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 3280.672191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3280.673718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.674882] Lustre: Skipped 12 previous similar messages [ 3280.679517] Lustre: Skipped 5 previous similar messages [ 3283.168189] Lustre: server umount lustre-OST0000 complete [ 3283.170533] Lustre: Skipped 3 previous similar messages [ 3291.103907] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.107674] LustreError: Skipped 1 previous similar message [ 3291.778897] LustreError: 222794:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325058 with bad export cookie 15959907962913456604 [ 3291.781423] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3291.782030] LustreError: 222794:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3294.643059] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:57:41 (1761325061) [ 3296.808076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.357519] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3301.229770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.883221] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3305.523786] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3307.515616] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3309.748198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3311.238266] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3313.860789] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3315.450431] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3315.546116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3317.046218] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3317.130154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3319.422817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3319.449093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3319.451797] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3319.498218] Lustre: lustre-OST0001: new disk, initializing [ 3319.500144] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3320.884813] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3324.763378] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3326.203814] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3326.970041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3326.972664] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3327.301958] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3328.854571] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3328.950473] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3332.066962] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3332.069205] LustreError: Skipped 3 previous similar messages [ 3337.784651] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3337.860193] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3339.324806] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3339.396138] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3357.665449] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.670462] LustreError: Skipped 15 previous similar messages [ 3362.923484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3364.377412] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3366.824724] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3367.352928] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3400.964960] LustreError: 232559:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325167 with bad export cookie 15959907962913457773 [ 3400.968069] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3400.969654] LustreError: 232559:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3409.096874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3410.379138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3411.634294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3412.923604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3415.060374] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:59:42 (1761325182) [ 3417.333052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3417.427835] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3417.438050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3417.510458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3417.513302] Lustre: Skipped 5 previous similar messages [ 3418.966895] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3421.766845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.797744] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3423.357915] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3425.927754] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3427.913099] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3430.391761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3430.458295] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3431.847223] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3434.338531] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3443.682277] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3443.710869] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3445.227600] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3447.749291] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3450.300359] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3451.683093] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3451.750065] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3453.397235] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3453.482645] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3455.557249] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff901d08815000.ost_server_uuid 40 [ 3456.110905] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff901d08815000.ost_server_uuid in FULL state after 0 sec [ 3456.464025] Lustre: Permanently deactivating lustre-OST0001 [ 3456.469298] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 3456.471740] Lustre: Skipped 1 previous similar message [ 3457.310190] Lustre: Permanently reactivating lustre-OST0001 [ 3460.065935] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.069407] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3460.069891] Lustre: Skipped 8 previous similar messages [ 3460.072269] LustreError: Skipped 3 previous similar messages [ 3460.073154] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3460.078648] Lustre: Skipped 22 previous similar messages [ 3464.375495] Lustre: server umount lustre-OST0001 complete [ 3464.378086] Lustre: Skipped 7 previous similar messages [ 3480.433463] LustreError: 239905:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325247 with bad export cookie 15959907962913459019 [ 3480.436879] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3480.438675] LustreError: 239905:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3482.669421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3484.152797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3485.914519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3487.460726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3489.734209] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 13:00:57 (1761325257) [ 3490.067968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3492.342254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.442139] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3492.450642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3492.452619] Lustre: Skipped 5 previous similar messages [ 3493.942231] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3496.809325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3496.840961] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3498.301297] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3500.754796] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3502.768038] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3505.100739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3505.132151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3505.205725] Lustre: Permanently deactivating lustre-OST0000 [ 3506.632632] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3507.889095] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3507.893223] LustreError: 247205:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 3509.199262] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3511.439532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3512.912715] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3515.420678] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3517.978749] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 3521.299697] Lustre: Permanently reactivating lustre-OST0000 [ 3528.195640] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3528.198572] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3528.202297] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 3528.219636] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3528.225762] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3538.403121] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.409311] LustreError: Skipped 15 previous similar messages [ 3550.449028] LustreError: 247730:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325317 with bad export cookie 15959907962913460608 [ 3550.452080] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3550.452590] LustreError: 247730:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3555.546523] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3556.509270] Key type lgssc unregistered [ 3556.607320] LNet: 249718:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3556.609630] LNet: Removed LNI 192.168.204.123@tcp [ 3556.896418] Key type .llcrypt unregistered [ 3556.897871] Key type ._llcrypt unregistered [ 3565.481053] alg: No test for adler32 (adler32-zlib) [ 3566.232469] Key type ._llcrypt registered [ 3566.234289] Key type .llcrypt registered [ 3566.275947] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3566.808942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3567.011363] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3567.104177] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3567.105826] LNet: Accept secure, port 988 [ 3568.695082] Key type lgssc registered [ 3569.089262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3570.273410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3572.493595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.673733] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3573.693532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3575.165989] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3577.990632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3578.015416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.048929] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3578.059243] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3578.061755] Lustre: Permanently deactivating lustre-MDT0001 [ 3578.186246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3579.627433] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3581.545636] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3582.129958] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3584.106250] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3586.375073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3586.477085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3587.837044] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3589.538911] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3589.541791] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 3590.436549] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3592.692533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3592.759181] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3593.769782] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:67 to 0x2c0000400:97 [ 3593.770565] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 3594.327104] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3596.917163] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3598.580465] Lustre: Permanently reactivating lustre-MDT0001 [ 3598.586741] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 3598.589875] Lustre: Skipped 3 previous similar messages [ 3601.980148] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3601.984718] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3601.989143] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3601.992694] LustreError: 254824:0:(llog_osd.c:2141:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3601.993335] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 3601.996680] LustreError: 254824:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3602.004346] LustreError: 254824:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.123@tcp: cfg command failed: rc = -108 [ 3602.009576] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3602.009576] [ 3602.014235] LustreError: 251221:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3604.327085] Lustre: lustre-MDT0001: Client 69373453-9ace-4013-bd49-af9cd65f306d (at 192.168.204.23@tcp) reconnecting [ 3606.659661] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3607.234829] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3609.111623] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 3609.193245] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3610.731382] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 3610.811455] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3611.615600] Lustre: Permanently deactivating lustre-MDT0001 [ 3611.621240] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 3611.624324] Lustre: Skipped 2 previous similar messages [ 3620.856413] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3629.539522] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.542035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3629.543217] Lustre: Skipped 3 previous similar messages [ 3629.545026] Lustre: Skipped 2 previous similar messages [ 3629.547262] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3634.657558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.660208] Lustre: Skipped 4 previous similar messages [ 3634.702900] Lustre: server umount lustre-MDT0000 complete [ 3636.037166] LustreError: 251210:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325403 with bad export cookie 15467559361596581271 [ 3636.039814] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3636.040808] LustreError: 251210:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3636.180031] Lustre: server umount lustre-MDT0001 complete [ 3643.632308] Lustre: server umount lustre-OST0000 complete [ 3647.512793] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 13:03:34 (1761325414) [ 3651.171400] Lustre: server umount lustre-OST0001 complete [ 3653.392843] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3656.529840] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3660.350759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3662.609204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3664.622351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3666.766284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3669.072514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3669.099425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3669.192144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3669.195673] Lustre: Skipped 2 previous similar messages [ 3669.208250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3669.245825] Lustre: lustre-MDT0000: new disk, initializing [ 3669.270689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3669.276658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3670.738540] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3674.757800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3674.777803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3674.809793] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3674.824599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3674.826133] Lustre: Skipped 1 previous similar message [ 3674.863112] Lustre: lustre-MDT0001: new disk, initializing [ 3674.898445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3674.900779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3676.261995] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3680.047962] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3681.926868] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3684.176302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3684.203912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3684.208703] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3684.322713] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3684.329642] Lustre: lustre-OST0000: new disk, initializing [ 3684.331288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3684.357439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3684.359323] Lustre: Skipped 1 previous similar message [ 3685.699844] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3689.478902] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3690.158547] LustreError: 259724:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3694.382436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3694.406367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3694.409946] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3694.462590] Lustre: lustre-OST0001: new disk, initializing [ 3694.465516] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3694.488098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3694.491179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3695.913762] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3699.688964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3699.690279] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3699.691463] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3700.211938] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3700.247051] LustreError: 259724:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3700.253949] LustreError: 259724:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3710.343068] LustreError: 259724:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3715.042199] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3715.044523] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3715.044598] LustreError: Skipped 1 previous similar message [ 3715.047860] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3715.049723] Lustre: Skipped 1 previous similar message [ 3717.239780] Lustre: server umount lustre-OST0001 complete [ 3720.161666] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3720.166401] LustreError: Skipped 1 previous similar message [ 3725.284859] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3725.288490] LustreError: Skipped 1 previous similar message [ 3730.401133] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3730.401608] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3730.402033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.402051] Lustre: Skipped 1 previous similar message [ 3730.402836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3730.402842] Lustre: Skipped 1 previous similar message [ 3730.403617] LustreError: Skipped 1 previous similar message [ 3730.406926] LustreError: Skipped 1 previous similar message [ 3735.331683] Lustre: server umount lustre-OST0000 complete [ 3740.641027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3740.641909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3740.644235] Lustre: Skipped 2 previous similar messages [ 3740.646779] Lustre: Skipped 2 previous similar messages [ 3744.079951] LustreError: 259710:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325511 with bad export cookie 15467559361596583819 [ 3744.082974] LustreError: 259710:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3744.083487] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3755.136753] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3756.097961] Key type lgssc unregistered [ 3756.209268] LNet: 264930:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3756.211531] LNet: Removed LNI 192.168.204.123@tcp [ 3756.489932] Key type .llcrypt unregistered [ 3756.490954] Key type ._llcrypt unregistered [ 3767.440332] alg: No test for adler32 (adler32-zlib) [ 3768.191315] Key type ._llcrypt registered [ 3768.192573] Key type .llcrypt registered [ 3768.229675] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 3774.095547] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3774.514066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3774.643336] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3774.714603] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3774.715957] LNet: Accept secure, port 988 [ 3776.295122] Key type lgssc registered [ 3776.596514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3779.506186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3781.617066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3783.694232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3785.790220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3788.051833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3788.082581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3789.199719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3789.210841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3789.259167] Lustre: lustre-MDT0000: new disk, initializing [ 3789.286721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3789.293883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3790.763828] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3794.904502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3794.933037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3794.965208] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3794.983131] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3794.985429] Lustre: Skipped 1 previous similar message [ 3795.035869] Lustre: lustre-MDT0001: new disk, initializing [ 3795.066176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3795.077423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3795.079958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3796.482870] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3800.331169] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3802.238044] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3804.656513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3804.686270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3804.688802] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3804.780996] Lustre: lustre-OST0000: new disk, initializing [ 3804.782594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3804.806661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.208614] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3810.212450] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3811.867306] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3814.902815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3814.905474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3815.076654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3816.856069] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3816.935433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3820.004880] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3820.004995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.006238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3820.009389] LustreError: Skipped 1 previous similar message [ 3820.016255] Lustre: Skipped 1 previous similar message [ 3823.776677] Lustre: server umount lustre-OST0000 complete [ 3825.120045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3825.120495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3825.122968] Lustre: Skipped 1 previous similar message [ 3825.125255] Lustre: Skipped 2 previous similar messages [ 3830.239880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3830.241608] Lustre: Skipped 1 previous similar message [ 3831.246441] Lustre: server umount lustre-MDT0000 complete [ 3832.627432] LustreError: 269014:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325599 with bad export cookie 5512949772873760926 [ 3832.630957] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3832.631636] LustreError: 269014:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3832.776224] Lustre: server umount lustre-MDT0001 complete [ 3835.826762] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 13:06:43 (1761325603) [ 3837.942929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3838.101431] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3838.124266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3839.490548] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3842.319057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.426973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3843.866886] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3846.376872] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3848.231342] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3850.465553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3850.602672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3851.950351] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3854.397059] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3868.130460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3868.134267] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3868.134691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3868.137946] Lustre: Skipped 1 previous similar message [ 3870.209774] Lustre: server umount lustre-OST0000 complete [ 3871.868893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3876.093710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3876.213187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3877.570230] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3879.975312] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3884.384414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3889.263533] Lustre: server umount lustre-OST0000 complete [ 3893.727989] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3893.730393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3893.734017] Lustre: Skipped 1 previous similar message [ 3893.736044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3893.737603] Lustre: Skipped 1 previous similar message [ 3897.927231] LustreError: 273577:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325664 with bad export cookie 5512949772873761766 [ 3897.930756] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3897.931195] LustreError: 273577:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3898.082145] Lustre: server umount lustre-MDT0001 complete [ 3898.084287] Lustre: Skipped 1 previous similar message [ 3902.874238] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3903.757962] Key type lgssc unregistered [ 3903.864376] LNet: 279004:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3903.866607] LNet: Removed LNI 192.168.204.123@tcp [ 3904.171548] Key type .llcrypt unregistered [ 3904.172671] Key type ._llcrypt unregistered [ 3912.788682] alg: No test for adler32 (adler32-zlib) [ 3913.541486] Key type ._llcrypt registered [ 3913.542546] Key type .llcrypt registered [ 3913.583304] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3914.078805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3914.240135] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3914.327240] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3914.328957] LNet: Accept secure, port 988 [ 3915.911158] Key type lgssc registered [ 3916.293661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3918.982873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3920.162714] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3920.178619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3921.559529] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3924.439195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3924.546448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3926.031455] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3928.464993] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3930.290989] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3932.492843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3932.612952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.081092] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3935.717833] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3936.441142] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3949.025360] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3949.029075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.034941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3950.049675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3950.049737] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3950.054021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3954.594797] Lustre: server umount lustre-OST0000 complete [ 3960.288650] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3960.288790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3960.289249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3960.289253] Lustre: Skipped 1 previous similar message [ 3960.299190] Lustre: Skipped 1 previous similar message [ 3961.953277] Lustre: server umount lustre-MDT0000 complete [ 3963.274022] LustreError: 284086:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325730 with bad export cookie 12024511331018013427 [ 3963.276890] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3963.278437] LustreError: 284086:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3963.405374] Lustre: server umount lustre-MDT0001 complete [ 3968.220393] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 3969.216294] Key type lgssc unregistered [ 3969.327399] LNet: 284889:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3969.331553] LNet: Removed LNI 192.168.204.123@tcp [ 3969.620844] Key type .llcrypt unregistered [ 3969.621826] Key type ._llcrypt unregistered [ 3976.349670] alg: No test for adler32 (adler32-zlib) [ 3977.101483] Key type ._llcrypt registered [ 3977.102958] Key type .llcrypt registered [ 3977.138729] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 3977.529460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3977.681303] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3977.754080] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 3977.755709] LNet: Accept secure, port 988 [ 3979.327054] Key type lgssc registered [ 3979.658658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3982.350920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.507745] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3983.522967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3984.865389] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3987.929861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3988.048575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3989.516604] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3992.054622] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3993.991178] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3996.208065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3996.348459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3997.706733] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 3999.464878] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 4000.119158] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4008.928895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.929098] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4008.933820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4008.937094] Lustre: Skipped 1 previous similar message [ 4014.048857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4014.050848] Lustre: Skipped 1 previous similar message [ 4014.829151] Lustre: server umount lustre-OST0000 complete [ 4018.656076] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4018.658108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.661045] Lustre: Skipped 1 previous similar message [ 4018.662519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4022.146794] Lustre: server umount lustre-MDT0000 complete [ 4023.384559] LustreError: 288128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325790 with bad export cookie 4905879148605626826 [ 4023.390129] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.392094] LustreError: 288128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4023.556840] Lustre: server umount lustre-MDT0001 complete [ 4028.286880] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4029.345624] Key type lgssc unregistered [ 4029.465440] LNet: 289975:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4029.468193] LNet: Removed LNI 192.168.204.123@tcp [ 4029.784966] Key type .llcrypt unregistered [ 4029.786338] Key type ._llcrypt unregistered [ 4036.495172] alg: No test for adler32 (adler32-zlib) [ 4037.246355] Key type ._llcrypt registered [ 4037.247397] Key type .llcrypt registered [ 4037.288131] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4037.702813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4038.006479] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4038.089272] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4038.090836] LNet: Accept secure, port 988 [ 4039.679089] Key type lgssc registered [ 4040.057914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4042.690704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4043.853384] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.869711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4045.199323] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4047.933667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.063632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4049.558140] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4052.039736] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4053.979878] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4056.261366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4056.394161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.885834] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4059.498176] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 4060.405957] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4068.835486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4068.836554] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4068.836820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4068.838767] Lustre: Skipped 1 previous similar message [ 4073.769250] Lustre: server umount lustre-OST0000 complete [ 4079.072603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.072834] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4079.073327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4079.073332] Lustre: Skipped 1 previous similar message [ 4079.075591] Lustre: Skipped 1 previous similar message [ 4081.220970] Lustre: server umount lustre-MDT0000 complete [ 4082.465253] LustreError: 291221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325849 with bad export cookie 8538177337736638244 [ 4082.467173] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4082.469661] LustreError: 291221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4082.615452] Lustre: server umount lustre-MDT0001 complete [ 4087.483259] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4088.513914] Key type lgssc unregistered [ 4088.617358] LNet: 294892:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4088.621237] LNet: Removed LNI 192.168.204.123@tcp [ 4088.963654] Key type .llcrypt unregistered [ 4088.965658] Key type ._llcrypt unregistered [ 4098.314134] alg: No test for adler32 (adler32-zlib) [ 4099.067357] Key type ._llcrypt registered [ 4099.068970] Key type .llcrypt registered [ 4099.114589] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4099.622678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4099.907569] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4100.001832] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4100.003409] LNet: Accept secure, port 988 [ 4101.591090] Key type lgssc registered [ 4101.944729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4104.680737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4105.836721] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4105.852034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4107.255949] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4110.049440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4110.165602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4111.609864] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4113.963519] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4115.824417] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4118.079626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4118.199127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4119.619295] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4121.253761] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 4122.107734] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4134.881371] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4134.884248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.888893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4135.904853] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4135.905164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4135.907556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4140.905559] Lustre: server umount lustre-OST0000 complete [ 4146.144652] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4146.145260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.146062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4146.146069] Lustre: Skipped 1 previous similar message [ 4146.152959] Lustre: Skipped 1 previous similar message [ 4148.312240] Lustre: server umount lustre-MDT0000 complete [ 4149.652146] LustreError: 298299:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325916 with bad export cookie 17804634941034719485 [ 4149.655852] LustreError: 298299:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4149.656195] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4149.815432] Lustre: server umount lustre-MDT0001 complete [ 4154.535691] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4155.559788] Key type lgssc unregistered [ 4155.669272] LNet: 300824:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4155.671791] LNet: Removed LNI 192.168.204.123@tcp [ 4155.953888] Key type .llcrypt unregistered [ 4155.955550] Key type ._llcrypt unregistered [ 4162.738520] alg: No test for adler32 (adler32-zlib) [ 4163.490359] Key type ._llcrypt registered [ 4163.491864] Key type .llcrypt registered [ 4163.531651] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4163.966448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4164.250723] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4164.329605] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4164.331291] LNet: Accept secure, port 988 [ 4165.911129] Key type lgssc registered [ 4166.265621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4168.825757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4170.015656] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.030457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.431905] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4174.228694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.345594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4175.744840] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4178.294255] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4180.209404] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4182.425394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4182.544449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4183.939680] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4185.636446] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 4186.414423] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4194.784421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.784866] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4194.788130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4194.791668] Lustre: Skipped 1 previous similar message [ 4199.906155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4199.907802] Lustre: Skipped 1 previous similar message [ 4200.745052] Lustre: server umount lustre-OST0000 complete [ 4205.024640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.024798] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4205.025059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.029623] Lustre: Skipped 2 previous similar messages [ 4210.144560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4210.145982] Lustre: Skipped 2 previous similar messages [ 4215.264051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4215.266636] Lustre: Skipped 1 previous similar message [ 4216.287115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4216.331288] Lustre: server umount lustre-MDT0000 complete [ 4217.586984] LustreError: 305120:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325984 with bad export cookie 14879698878109371434 [ 4217.590095] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4217.590540] LustreError: 305120:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4217.733296] Lustre: server umount lustre-MDT0001 complete [ 4222.358627] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4223.374644] Key type lgssc unregistered [ 4223.497412] LNet: 305924:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4223.500750] LNet: Removed LNI 192.168.204.123@tcp [ 4223.806826] Key type .llcrypt unregistered [ 4223.807792] Key type ._llcrypt unregistered [ 4230.139845] alg: No test for adler32 (adler32-zlib) [ 4230.891331] Key type ._llcrypt registered [ 4230.892260] Key type .llcrypt registered [ 4230.927929] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4231.292480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4231.556700] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4231.637647] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4231.640309] LNet: Accept secure, port 988 [ 4233.223185] Key type lgssc registered [ 4233.541062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4236.115282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4237.284857] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.299281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.673660] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4241.552136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4241.670995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4243.195483] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4245.603765] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4247.489653] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4249.813106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4249.930566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4251.372094] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4252.008216] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 4253.843386] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4261.344810] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4261.345219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.349603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4261.352047] Lustre: Skipped 1 previous similar message [ 4261.356577] Lustre: Skipped 1 previous similar message [ 4266.466177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4266.467786] Lustre: Skipped 1 previous similar message [ 4267.242936] Lustre: server umount lustre-OST0000 complete [ 4271.584291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4271.585075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4271.589832] Lustre: Skipped 1 previous similar message [ 4274.696841] Lustre: server umount lustre-MDT0000 complete [ 4275.962141] LustreError: 307169:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326042 with bad export cookie 10150100796894211272 [ 4275.965552] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4275.966455] LustreError: 307169:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4276.112319] Lustre: server umount lustre-MDT0001 complete [ 4280.789140] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4281.733780] Key type lgssc unregistered [ 4281.837354] LNet: 310844:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4281.840946] LNet: Removed LNI 192.168.204.123@tcp [ 4282.116571] Key type .llcrypt unregistered [ 4282.117787] Key type ._llcrypt unregistered [ 4286.625251] alg: No test for adler32 (adler32-zlib) [ 4287.377358] Key type ._llcrypt registered [ 4287.378368] Key type .llcrypt registered [ 4287.419251] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4293.233569] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 4298.555379] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4298.901404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4299.198711] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4299.288759] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4299.290620] LNet: Accept secure, port 988 [ 4300.871117] Key type lgssc registered [ 4301.244416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4304.316085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4306.551352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4308.631521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4310.669485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4312.866799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4312.891546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4313.995429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4314.008389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4314.054748] Lustre: lustre-MDT0000: new disk, initializing [ 4314.082037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4314.091273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4315.631158] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4319.792580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4319.816908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4319.849478] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4319.871264] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4319.873621] Lustre: Skipped 1 previous similar message [ 4319.917662] Lustre: lustre-MDT0001: new disk, initializing [ 4319.955499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4319.969293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4319.972931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4321.454911] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4325.235724] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4327.257729] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4329.553740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4329.585275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4329.587736] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4329.681246] Lustre: lustre-OST0000: new disk, initializing [ 4329.682961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4329.707846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4331.068514] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4334.842653] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4336.326897] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4336.628093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4336.630536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4337.417293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4339.107261] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4339.190827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4341.729435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4341.729789] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4341.735092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4341.739181] Lustre: Skipped 1 previous similar message [ 4341.923457] Lustre: server umount lustre-OST0000 complete [ 4345.823687] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4345.825649] LustreError: Skipped 1 previous similar message [ 4345.826686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.829305] Lustre: Skipped 1 previous similar message [ 4345.830650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4346.848166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4346.848670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.320205] Lustre: server umount lustre-MDT0000 complete [ 4350.618352] LustreError: 314581:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326117 with bad export cookie 12633093937895999555 [ 4350.620864] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4350.623651] LustreError: 314581:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4350.776315] Lustre: server umount lustre-MDT0001 complete [ 4353.499677] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:15:20 (1761326120) [ 4355.678789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4355.829976] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4355.851785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4357.174372] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4359.920933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4360.025788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4361.446224] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4364.016789] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4366.027474] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4368.283824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4368.389394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4369.756643] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4372.281704] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4380.642190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4380.644219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4380.645502] Lustre: Skipped 1 previous similar message [ 4380.648478] Lustre: Skipped 1 previous similar message [ 4380.649897] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4385.761277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4386.355959] Lustre: server umount lustre-OST0000 complete [ 4390.881721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4390.881851] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4390.885805] Lustre: Skipped 1 previous similar message [ 4393.865046] Lustre: server umount lustre-MDT0000 complete [ 4395.152542] LustreError: 319121:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326162 with bad export cookie 12633093937896000395 [ 4395.156067] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.157890] LustreError: 319121:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4400.096348] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4401.134155] Key type lgssc unregistered [ 4401.264352] LNet: 322795:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4401.267228] LNet: Removed LNI 192.168.204.123@tcp [ 4401.602339] Key type .llcrypt unregistered [ 4401.603522] Key type ._llcrypt unregistered [ 4411.015775] alg: No test for adler32 (adler32-zlib) [ 4411.767269] Key type ._llcrypt registered [ 4411.768248] Key type .llcrypt registered [ 4411.808699] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 4417.362816] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4417.787127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4417.926658] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4418.002744] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4418.004435] LNet: Accept secure, port 988 [ 4419.583125] Key type lgssc registered [ 4419.956449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4422.941502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4425.089707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4427.035671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4429.105622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4430.832653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4432.549897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4434.827934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4434.853458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4435.970642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4435.979917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4436.017545] Lustre: lustre-MDT0000: new disk, initializing [ 4436.043329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.051551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4437.521548] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4441.720713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4441.742691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4441.774627] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4441.790221] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4441.791783] Lustre: Skipped 1 previous similar message [ 4441.830226] Lustre: lustre-MDT0001: new disk, initializing [ 4441.861040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4441.873526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4441.878211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4443.281478] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4447.137711] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4448.998718] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4451.226767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4451.253822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4451.256107] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4451.376803] Lustre: lustre-OST2710: new disk, initializing [ 4451.378579] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 4451.407553] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4452.826458] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4456.654992] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4458.970941] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4459.001328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4459.002032] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4459.004979] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4459.007394] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4459.072827] Lustre: lustre-OST03e8: new disk, initializing [ 4459.074777] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 4459.103428] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4460.573738] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4463.241681] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4465.144465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4465.147798] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4466.091211] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 4466.174858] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4467.631717] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 4467.719121] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4469.223396] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 4469.309405] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4470.816260] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 4470.897991] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4472.799995] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4472.802953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.806404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4475.361220] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4475.361220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4475.362281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4475.364552] Lustre: Skipped 1 previous similar message [ 4475.367596] Lustre: Skipped 2 previous similar messages [ 4478.026308] Lustre: server umount lustre-MDT0000 complete [ 4479.227217] LustreError: 327583:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326246 with bad export cookie 6491468851057534921 [ 4479.230454] LustreError: 327583:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4479.231408] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4479.382405] Lustre: server umount lustre-MDT0001 complete [ 4486.111182] Lustre: 332215:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326247/real 1761326247] req@0000000016ecf658 x1846884365785472/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761326253 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4486.143980] Lustre: server umount lustre-OST2710 complete [ 4493.791126] Lustre: 332416:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326254/real 1761326254] req@00000000305907e6 x1846884365785792/t0(0) o39->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761326260 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4493.870320] Lustre: server umount lustre-OST03e8 complete [ 4499.779067] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 4502.624439] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4505.716881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4507.844793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4509.836942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4511.900259] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4514.035425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4514.059978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4514.139810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4514.150791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4514.197346] Lustre: lustre-MDT0000: new disk, initializing [ 4514.224782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4514.230879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4515.612088] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4519.718590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4519.743514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4519.773987] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4519.864390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4521.254400] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4524.979274] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4526.860811] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4529.037708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4529.064700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4529.068846] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4529.165894] Lustre: lustre-OST0000: new disk, initializing [ 4529.167583] Lustre: Skipped 1 previous similar message [ 4529.169506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4529.173294] Lustre: Skipped 2 previous similar messages [ 4529.197660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4529.199500] Lustre: Skipped 1 previous similar message [ 4529.444797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4529.447427] Lustre: Skipped 1 previous similar message [ 4529.449289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4530.644714] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4534.313308] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4535.808254] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4535.888502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4537.265159] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4537.336293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4539.875120] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4539.876490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.880742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4539.882279] Lustre: Skipped 1 previous similar message [ 4544.099193] Lustre: server umount lustre-OST0000 complete [ 4545.503906] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4545.505853] LustreError: Skipped 1 previous similar message [ 4545.506995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.509745] Lustre: Skipped 1 previous similar message [ 4545.511199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4550.112205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4550.113770] Lustre: Skipped 1 previous similar message [ 4552.724291] LustreError: 336170:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326319 with bad export cookie 6491468851057536517 [ 4552.726551] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4552.726895] LustreError: 336170:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4552.861054] Lustre: server umount lustre-MDT0001 complete [ 4552.862306] Lustre: Skipped 1 previous similar message [ 4555.495526] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:18:42 (1761326322) [ 4556.011178] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4556.529898] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:18:43 (1761326323) [ 4557.576200] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4560.077454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.299987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4562.518590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4563.750453] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4564.094183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4566.267922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.367150] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4566.375853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4566.432725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.708374] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4570.399557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4570.430049] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4571.816341] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4574.248211] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4576.088133] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4578.261194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4578.287098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4578.350845] Lustre: Denying initial registration attempt from nid 192.168.204.123@tcp, specified as failover [ 4578.353811] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4578.356329] LustreError: 343334:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 4578.358316] LustreError: 343334:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 4578.360083] LustreError: 343334:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 4578.403628] Lustre: server umount lustre-OST0000 complete [ 4578.404898] LustreError: 343334:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4580.831871] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4580.834054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.837403] Lustre: Skipped 1 previous similar message [ 4580.839548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4586.464351] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.470559] LustreError: Skipped 1 previous similar message [ 4587.256961] LustreError: 342177:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326354 with bad export cookie 6491468851057537357 [ 4587.258951] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4587.262399] LustreError: 342177:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4591.957204] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4592.914788] Key type lgssc unregistered [ 4593.031469] LNet: 344407:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4593.033819] LNet: Removed LNI 192.168.204.123@tcp [ 4593.305753] Key type .llcrypt unregistered [ 4593.307204] Key type ._llcrypt unregistered [ 4597.426099] alg: No test for adler32 (adler32-zlib) [ 4598.177415] Key type ._llcrypt registered [ 4598.178666] Key type .llcrypt registered [ 4598.214889] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4598.595742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4598.765170] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4598.839563] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4598.841023] LNet: Accept secure, port 988 [ 4600.415087] Key type lgssc registered [ 4600.741706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4603.016730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4604.247079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4605.483167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4606.705847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4607.052861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.500591] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4615.513358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.624417] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4616.631636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4616.686779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4618.172404] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4620.858645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4620.891917] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4620.902471] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4620.980406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4622.322689] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4624.613308] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4626.346592] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4628.449354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4628.477270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4628.556871] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4628.614044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4630.006618] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4632.309174] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4633.031708] Lustre: server umount lustre-OST0000 complete [ 4636.639917] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4636.642810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.647061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4638.688496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4638.689399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4640.326417] Lustre: server umount lustre-MDT0000 complete [ 4641.529212] LustreError: 349255:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326408 with bad export cookie 11541495489105769067 [ 4641.532873] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4641.534873] LustreError: 349255:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4641.686201] Lustre: server umount lustre-MDT0001 complete [ 4644.923233] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4645.959420] Key type lgssc unregistered [ 4646.081534] LNet: 350967:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4646.084435] LNet: Removed LNI 192.168.204.123@tcp [ 4646.372960] Key type .llcrypt unregistered [ 4646.374909] Key type ._llcrypt unregistered [ 4654.638530] alg: No test for adler32 (adler32-zlib) [ 4655.389306] Key type ._llcrypt registered [ 4655.390276] Key type .llcrypt registered [ 4655.424718] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4655.816301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4656.008545] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4656.087740] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4656.089255] LNet: Accept secure, port 988 [ 4657.671095] Key type lgssc registered [ 4658.048945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4660.610044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4661.788137] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4661.804322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.155948] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4665.837730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.977328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4667.303824] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4669.591105] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4671.359575] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4673.606136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4673.725547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4675.201270] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4677.554082] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4685.691647] LustreError: 355076:0:(ldlm_resource.c:1127:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200001b71:0x1:0x0].0xf14dc48e (00000000e96d7fdf) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4686.815818] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4686.817882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.821643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4688.353669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4688.354266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.358253] Lustre: Skipped 1 previous similar message [ 4691.853405] Lustre: server umount lustre-MDT0000 complete [ 4693.085261] LustreError: 352409:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326460 with bad export cookie 14720587454438076222 [ 4693.089409] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4693.472074] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4693.472191] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4693.479411] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4693.480855] Lustre: Skipped 3 previous similar messages [ 4698.591949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4698.592049] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4698.594556] Lustre: Skipped 1 previous similar message [ 4698.601251] LustreError: Skipped 1 previous similar message [ 4699.298390] Lustre: server umount lustre-MDT0001 complete [ 4701.280826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4703.996277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4710.880763] Lustre: Evicted from MGS (at 192.168.204.123@tcp) after server handle changed from 0xcc4a084bfada6f3e to 0xcc4a084bfada8b1b [ 4710.884227] Lustre: MGC192.168.204.123@tcp: Connection restored to 192.168.204.123@tcp (at 0@lo) [ 4710.992227] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4711.017957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4711.044779] Lustre: lustre-OST0000: deleting orphan objects from 0x0:134 to 0x0:161 [ 4712.613411] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4715.104662] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4715.107743] LustreError: Skipped 2 previous similar messages [ 4715.312675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4715.432368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4716.839332] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4719.223558] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4720.547561] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4721.807717] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4723.606373] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4728.148501] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4730.849174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.849746] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4730.851381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4730.852264] Lustre: Skipped 1 previous similar message [ 4735.403559] Lustre: server umount lustre-OST0000 complete [ 4741.089200] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4741.089712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4741.090406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4741.090411] Lustre: Skipped 1 previous similar message [ 4741.098784] Lustre: Skipped 1 previous similar message [ 4742.794751] Lustre: server umount lustre-MDT0000 complete [ 4743.961290] LustreError: 352408:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326510 with bad export cookie 14720587454438083355 [ 4743.964588] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4743.964732] LustreError: 352408:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4748.545372] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4749.586767] Key type lgssc unregistered [ 4749.693482] LNet: 358918:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4749.695851] LNet: Removed LNI 192.168.204.123@tcp [ 4749.974820] Key type .llcrypt unregistered [ 4749.975757] Key type ._llcrypt unregistered [ 4758.189980] alg: No test for adler32 (adler32-zlib) [ 4758.941342] Key type ._llcrypt registered [ 4758.942467] Key type .llcrypt registered [ 4758.976363] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4759.358970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4759.498515] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4759.566419] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4759.567918] LNet: Accept secure, port 988 [ 4761.143090] Key type lgssc registered [ 4761.455026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4763.764510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.915765] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4764.930113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4766.188111] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4768.814607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4768.920942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4770.199563] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4772.495191] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4774.219587] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4779.489084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4779.489162] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4779.490041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4779.492067] Lustre: Skipped 1 previous similar message [ 4781.257308] Lustre: server umount lustre-MDT0000 complete [ 4782.404947] LustreError: 361163:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326549 with bad export cookie 13943857320543537754 [ 4782.407108] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4782.407815] LustreError: 361163:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4782.523074] Lustre: server umount lustre-MDT0001 complete [ 4786.955726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.045902] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4787.054572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4787.116527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.339214] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4790.916260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.944511] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4790.952305] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4791.025877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4792.272252] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4794.394770] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4796.129974] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4798.417729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4798.489321] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4798.492215] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4798.496737] LustreError: 365069:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 4798.500364] LustreError: 365069:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 4798.503733] LustreError: 365069:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 4798.558031] Lustre: server umount lustre-OST0000 complete [ 4798.559337] LustreError: 365069:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4800.619199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4800.684974] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4800.742332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4801.962050] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4804.186813] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4806.648309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4806.676810] Lustre: lustre-OST0000: deleting orphan objects from 0x0:134 to 0x0:193 [ 4806.677223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4806.681396] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4806.709057] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4806.731841] Lustre: lustre-OST0001: new disk, initializing [ 4806.733451] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4807.949334] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4810.250291] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4811.177314] Lustre: server umount lustre-OST0001 complete [ 4813.292681] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4813.295952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.297753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4813.298875] Lustre: Skipped 1 previous similar message [ 4813.301736] Lustre: Skipped 1 previous similar message [ 4818.402306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4818.403589] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4818.405853] Lustre: Skipped 1 previous similar message [ 4818.411881] LustreError: Skipped 1 previous similar message [ 4818.486234] Lustre: server umount lustre-OST0000 complete [ 4821.983857] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4821.986033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4821.989360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4828.640358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4828.641990] Lustre: Skipped 3 previous similar messages [ 4834.271084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4834.309985] Lustre: server umount lustre-MDT0000 complete [ 4835.473097] LustreError: 363869:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326602 with bad export cookie 13943857320543538293 [ 4835.475342] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4835.478212] LustreError: 363869:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4839.200900] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 4840.201886] Key type lgssc unregistered [ 4840.317311] LNet: 368428:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4840.319771] LNet: Removed LNI 192.168.204.123@tcp [ 4840.598898] Key type .llcrypt unregistered [ 4840.600406] Key type ._llcrypt unregistered [ 4851.255771] alg: No test for adler32 (adler32-zlib) [ 4852.006286] Key type ._llcrypt registered [ 4852.007218] Key type .llcrypt registered [ 4852.051249] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 4857.523796] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4857.945396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4858.060930] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4858.130444] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 4858.131984] LNet: Accept secure, port 988 [ 4859.703107] Key type lgssc registered [ 4860.025109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4862.854936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4864.990180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4866.980101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4868.979343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4871.057485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4871.081423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.169312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4872.178487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4872.214254] Lustre: lustre-MDT0000: new disk, initializing [ 4872.235534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4872.240994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4873.492792] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4877.440688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4877.469540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4877.511293] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4877.543192] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4877.545059] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4877.546675] Lustre: Skipped 1 previous similar message [ 4877.582780] Lustre: lustre-MDT0001: new disk, initializing [ 4877.613887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4877.626370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4877.629040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4879.009318] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4882.618099] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4884.336107] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4886.522783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4886.547445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4886.549489] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4886.659290] Lustre: lustre-OST0000: new disk, initializing [ 4886.662323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4886.688328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4886.920043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4886.922927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4888.033169] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4891.585039] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4892.968046] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4893.039269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4894.394624] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4894.471628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4897.122633] Lustre: server umount lustre-OST0000 complete [ 4902.369173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4902.369657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4902.372111] Lustre: Skipped 1 previous similar message [ 4902.376769] Lustre: Skipped 1 previous similar message [ 4904.415130] Lustre: 370743:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326664/real 1761326664] req@000000009b83dc17 x1846884827148224/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761326671 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4904.415308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4904.422319] Lustre: 370743:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4904.426343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4904.428532] Lustre: Skipped 1 previous similar message [ 4904.486494] Lustre: server umount lustre-MDT0000 complete [ 4905.707257] LustreError: 374606:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326672 with bad export cookie 10254890948255072146 [ 4905.709820] LustreError: 374606:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4905.711718] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4905.849145] Lustre: server umount lustre-MDT0001 complete [ 4908.423800] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 13:24:35 (1761326675) [ 4910.204055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4911.977043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4918.760640] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 4921.241601] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 4924.043188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4925.975433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4927.853713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4929.796335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4931.924704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4931.950540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4932.052962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4932.062484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4932.101488] Lustre: lustre-MDT0000: new disk, initializing [ 4932.124962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4932.130465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4933.422060] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4937.347301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4937.375639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4937.405497] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4937.424044] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4937.427478] Lustre: Skipped 1 previous similar message [ 4937.467417] Lustre: lustre-MDT0001: new disk, initializing [ 4937.498934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4937.512840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4937.515435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4938.895823] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4942.552628] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4944.415707] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4946.553100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4946.582405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4946.584513] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4946.684964] Lustre: lustre-OST0000: new disk, initializing [ 4946.686503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4946.713546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4948.069965] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4951.732701] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4953.043534] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4954.610186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4954.612779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4955.159312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4956.435974] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4956.508281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4959.713026] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4959.713951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4959.714871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4959.718733] Lustre: Skipped 1 previous similar message [ 4963.234848] Lustre: server umount lustre-OST0000 complete [ 4964.832034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4964.835282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4964.836665] Lustre: Skipped 2 previous similar messages [ 4970.434923] Lustre: server umount lustre-MDT0000 complete [ 4971.575975] LustreError: 380268:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326738 with bad export cookie 10254890948255072972 [ 4971.578345] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4971.580077] LustreError: 380268:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4974.293385] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 13:25:41 (1761326741) [ 4976.782339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4976.930305] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.949610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4978.171346] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4980.684350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4982.069860] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4984.321081] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4986.089885] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4988.272827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4989.657095] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 4991.939461] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5000.247662] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 5000.727844] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 5001.215380] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 5001.719938] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 5002.326180] Lustre: Failing over lustre-MDT0000 [ 5002.503837] Lustre: server umount lustre-MDT0000 complete [ 5002.505167] Lustre: Skipped 1 previous similar message [ 5003.622358] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5005.793414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5005.796080] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5005.797690] Lustre: Skipped 3 previous similar messages [ 5005.800909] LustreError: Skipped 1 previous similar message [ 5008.736843] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5008.740683] LustreError: Skipped 1 previous similar message [ 5012.831146] Lustre: 370743:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326772/real 1761326772] req@0000000075d5bd73 x1846884827174976/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326779 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5012.836175] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5012.840121] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5012.843421] LustreError: Skipped 6 previous similar messages [ 5015.499941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5018.082698] Lustre: Evicted from MGS (at 192.168.204.123@tcp) after server handle changed from 0x8e50b0fc151cc614 to 0x8e50b0fc151cc9f6 [ 5018.086518] Lustre: MGC192.168.204.123@tcp: Connection restored to 192.168.204.123@tcp (at 0@lo) [ 5018.174102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5018.175668] Lustre: Skipped 2 previous similar messages [ 5018.201648] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5019.582691] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5022.049123] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5023.205591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5023.213721] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5023.230087] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5025.006568] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5025.545183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5027.336384] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 5027.924527] Lustre: Failing over lustre-MDT0000 [ 5028.108405] Lustre: server umount lustre-MDT0000 complete [ 5028.321238] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5028.321938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.323233] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5028.323238] LustreError: Skipped 5 previous similar messages [ 5028.323421] LustreError: Skipped 1 previous similar message [ 5028.326539] Lustre: Skipped 1 previous similar message [ 5035.487129] Lustre: 370744:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326795/real 1761326795] req@00000000fd71429f x1846884827182080/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326802 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5035.492344] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5041.060746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5041.636163] Lustre: Evicted from MGS (at 192.168.204.123@tcp) after server handle changed from 0x8e50b0fc151cc9f6 to 0x8e50b0fc151cccf8 [ 5041.639693] Lustre: MGC192.168.204.123@tcp: Connection restored to (at 0@lo) [ 5041.641149] Lustre: Skipped 2 previous similar messages [ 5041.738204] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5042.970895] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5044.577182] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5046.760838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5046.765117] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5046.779743] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 5048.432881] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5048.920209] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5051.873196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5051.873581] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5051.874142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5051.874149] Lustre: Skipped 2 previous similar messages [ 5051.876119] Lustre: Skipped 2 previous similar messages [ 5058.363133] LustreError: 384918:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326825 with bad export cookie 10254890948255075576 [ 5058.365960] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5058.366132] LustreError: 384918:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5065.695171] Lustre: 390719:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326826/real 1761326826] req@0000000006bc4001 x1846884827190912/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761326832 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5065.788597] Lustre: server umount lustre-OST0000 complete [ 5065.789757] Lustre: Skipped 2 previous similar messages [ 5070.456539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.617242] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5070.623885] LustreError: Skipped 17 previous similar messages [ 5072.064619] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5074.659974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5076.123407] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5078.406471] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5080.061931] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5082.062507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5083.494165] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5085.288509] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 5085.712219] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5094.881675] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5094.884087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.889988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5094.891407] Lustre: Skipped 6 previous similar messages [ 5106.687038] LustreError: 393496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326873 with bad export cookie 10254890948255076283 [ 5106.690065] LustreError: 393496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5106.690151] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5111.026338] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5112.015413] Key type lgssc unregistered [ 5112.132323] LNet: 395197:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5112.134804] LNet: Removed LNI 192.168.204.123@tcp [ 5112.436637] Key type .llcrypt unregistered [ 5112.437980] Key type ._llcrypt unregistered [ 5121.063304] alg: No test for adler32 (adler32-zlib) [ 5121.815369] Key type ._llcrypt registered [ 5121.816562] Key type .llcrypt registered [ 5121.848960] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5122.359512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5122.545460] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 5122.619918] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5122.621244] LNet: Accept secure, port 988 [ 5124.199070] Key type lgssc registered [ 5124.512493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5126.970947] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5126.973710] LustreError: 396680:0:(osd_handler.c:8259:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5126.981949] LustreError: 396680:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5126.983597] LustreError: 396680:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5126.985319] LustreError: 396680:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5126.987233] LustreError: 396680:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5129.324304] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5129.326636] LustreError: 397077:0:(osd_handler.c:8259:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5129.334078] LustreError: 397077:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5129.336013] LustreError: 397077:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5129.338039] LustreError: 397077:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5129.340360] LustreError: 397077:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5134.873843] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5135.899528] Key type lgssc unregistered [ 5136.022430] LNet: 398032:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5136.026627] LNet: Removed LNI 192.168.204.123@tcp [ 5136.326592] Key type .llcrypt unregistered [ 5136.327577] Key type ._llcrypt unregistered [ 5142.113941] alg: No test for adler32 (adler32-zlib) [ 5142.865315] Key type ._llcrypt registered [ 5142.866186] Key type .llcrypt registered [ 5142.900458] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 5148.230268] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5148.640110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5148.749745] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 5148.819152] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5148.820521] LNet: Accept secure, port 988 [ 5150.391107] Key type lgssc registered [ 5150.677205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5153.267653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5155.252486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5157.097430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5158.990623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5161.116747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5161.140931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5162.240549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5162.249856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5162.288050] Lustre: lustre-MDT0000: new disk, initializing [ 5162.311715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.317324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5163.656759] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5167.580986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5167.608623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5167.637329] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5167.654895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5167.656381] Lustre: Skipped 1 previous similar message [ 5167.691178] Lustre: lustre-MDT0001: new disk, initializing [ 5167.722517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5167.733221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5167.736220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5169.032036] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5172.617877] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5174.399565] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5176.523206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5176.547640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5176.550142] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5176.647789] Lustre: lustre-OST0000: new disk, initializing [ 5176.649471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5176.676978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5178.043811] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5181.624336] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5182.955630] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5186.546539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5186.548924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5187.107424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 5188.405671] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5188.473061] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5191.651256] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5191.651338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.651946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5191.653326] LustreError: Skipped 1 previous similar message [ 5191.656141] Lustre: Skipped 1 previous similar message [ 5195.174672] Lustre: server umount lustre-OST0000 complete [ 5196.768512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.769196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5196.771532] Lustre: Skipped 1 previous similar message [ 5196.773832] Lustre: Skipped 2 previous similar messages [ 5201.888844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5201.891739] Lustre: Skipped 1 previous similar message [ 5202.439148] Lustre: server umount lustre-MDT0000 complete [ 5203.676039] LustreError: 402159:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326970 with bad export cookie 14760925533580191972 [ 5203.679211] LustreError: 402159:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5203.680200] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5203.814599] Lustre: server umount lustre-MDT0001 complete [ 5206.274243] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5206.740983] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:29:34 (1761326974) [ 5208.725422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5208.879418] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.901897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5210.066699] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5212.621222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.079067] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5216.392454] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5218.024180] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5220.018314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5220.120776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5220.122322] Lustre: Skipped 1 previous similar message [ 5221.433425] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5223.773870] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5225.807819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5225.835990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5225.839180] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5225.895317] Lustre: lustre-OST0001: new disk, initializing [ 5225.898037] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5227.209012] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5228.861477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5228.864082] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5230.651782] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5234.144854] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5234.147168] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5234.151440] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5234.152834] Lustre: Skipped 1 previous similar message [ 5238.516297] Lustre: server umount lustre-OST0001 complete [ 5239.266792] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5239.270626] LustreError: Skipped 1 previous similar message [ 5244.389853] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5244.393892] LustreError: Skipped 1 previous similar message [ 5249.508340] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5249.511236] LustreError: Skipped 1 previous similar message [ 5252.575837] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5252.578367] LustreError: Skipped 1 previous similar message [ 5252.579569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5252.583552] Lustre: Skipped 1 previous similar message [ 5252.585322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5252.586664] Lustre: Skipped 1 previous similar message [ 5254.626838] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5254.631268] LustreError: Skipped 1 previous similar message [ 5256.482728] Lustre: server umount lustre-OST0000 complete [ 5258.719857] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5258.721729] LustreError: Skipped 1 previous similar message [ 5258.722764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5258.725436] Lustre: Skipped 1 previous similar message [ 5264.865222] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5264.868251] LustreError: Skipped 1 previous similar message [ 5264.978277] LustreError: 408834:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327032 with bad export cookie 14760925533580192812 [ 5264.981252] LustreError: 408834:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5264.981541] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5271.188558] Lustre: server umount lustre-MDT0001 complete [ 5271.189729] Lustre: Skipped 1 previous similar message [ 5275.433453] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5276.389275] Key type lgssc unregistered [ 5276.491439] LNet: 410832:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5276.493844] LNet: Removed LNI 192.168.204.123@tcp [ 5276.787644] Key type .llcrypt unregistered [ 5276.788613] Key type ._llcrypt unregistered [ 5287.149713] alg: No test for adler32 (adler32-zlib) [ 5287.900335] Key type ._llcrypt registered [ 5287.901211] Key type .llcrypt registered [ 5287.938761] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 5293.173943] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5293.594795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5293.704460] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 5293.773458] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5293.775641] LNet: Accept secure, port 988 [ 5295.351091] Key type lgssc registered [ 5295.638198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5298.712891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5301.299573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5303.871878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5306.098163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5308.432957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5308.456071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5309.564592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5309.573468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5309.610051] Lustre: lustre-MDT0000: new disk, initializing [ 5309.632427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5309.637896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5310.873251] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5314.599666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5314.628249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5314.667583] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5314.686353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5314.687780] Lustre: Skipped 1 previous similar message [ 5314.723153] Lustre: lustre-MDT0001: new disk, initializing [ 5314.758840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5314.773861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5314.778151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5316.107060] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5319.768283] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5321.729652] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5324.029940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5324.055389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5324.060287] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5324.178668] Lustre: lustre-OST0000: new disk, initializing [ 5324.180555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5324.208782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5325.740919] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5326.967119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5326.969955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5329.664281] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5331.334923] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5331.418405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5332.811114] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5332.895488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5337.568782] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5337.570929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5337.571522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5337.573889] Lustre: Skipped 1 previous similar message [ 5337.576285] Lustre: Skipped 1 previous similar message [ 5339.628795] Lustre: server umount lustre-OST0000 complete [ 5342.688397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5342.688917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5342.691458] Lustre: Skipped 1 previous similar message [ 5342.693741] Lustre: Skipped 1 previous similar message [ 5347.807880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5347.809603] Lustre: Skipped 1 previous similar message [ 5352.927839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5352.929405] Lustre: Skipped 1 previous similar message [ 5354.975106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5355.011509] Lustre: server umount lustre-MDT0000 complete [ 5356.113321] LustreError: 414913:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327123 with bad export cookie 8765223852101033521 [ 5356.116813] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5356.251795] Lustre: server umount lustre-MDT0001 complete [ 5358.687298] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 13:32:06 (1761327126) [ 5361.514744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5364.009876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5364.143332] Lustre: lustre-MDT0000: new disk, initializing [ 5364.152821] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5364.170795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5365.358401] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5367.886532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5369.211171] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5371.371389] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5373.014766] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5378.528348] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5378.528426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.528930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.528934] Lustre: Skipped 2 previous similar messages [ 5378.530188] LustreError: Skipped 1 previous similar message [ 5378.533024] Lustre: Skipped 1 previous similar message [ 5379.712780] Lustre: server umount lustre-MDT0000 complete [ 5380.821995] LustreError: 420046:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327147 with bad export cookie 8765223852101034361 [ 5380.824294] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5380.824641] LustreError: 420046:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5383.597439] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:32:30 (1761327150) [ 5385.568136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.716396] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.735155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.736716] Lustre: Skipped 1 previous similar message [ 5386.938436] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5389.556042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5390.976344] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5393.162868] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5394.797429] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5396.732826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5396.838593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5396.840125] Lustre: Skipped 1 previous similar message [ 5398.005299] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5400.110601] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5408.876311] Lustre: Permanently deactivating lustre-OST0000 [ 5408.880471] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 5414.383376] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5415.665224] LustreError: 425752:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 5415.668038] LustreError: 425752:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 5415.669890] LustreError: 425752:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 5416.203815] LustreError: 425801:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 5416.205639] LustreError: 425801:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 5422.888518] Lustre: server umount lustre-OST0000 complete [ 5422.889674] Lustre: Skipped 1 previous similar message [ 5423.689705] LustreError: 426050:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 5423.691553] LustreError: 426050:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 5424.608098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.608487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5424.611733] Lustre: Skipped 1 previous similar message [ 5424.613080] Lustre: Skipped 1 previous similar message [ 5430.751085] Lustre: 426147:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761327191/real 1761327191] req@00000000def3db18 x1846885283302080/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761327197 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5431.947838] LustreError: 424610:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327198 with bad export cookie 8765223852101034886 [ 5431.950033] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5431.951534] LustreError: 424610:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5437.407097] Lustre: 426350:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761327198/real 1761327198] req@0000000099dc5440 x1846885283303040/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761327204 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5437.537041] Lustre: server umount lustre-MDT0001 complete [ 5437.538278] Lustre: Skipped 1 previous similar message [ 5440.117672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5441.387065] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5442.212819] LustreError: 427257:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 5444.073831] Lustre: 427602:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.123@tcp [ 5450.719149] Lustre: 427703:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761327211/real 1761327211] req@0000000099dc5440 x1846885283303360/t0(0) o251->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761327217 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5454.230050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5454.397475] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.401709] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5454.404379] Lustre: Skipped 1 previous similar message [ 5454.422962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5455.783858] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5458.497886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5458.598928] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5460.162062] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5462.409936] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5464.197946] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5464.776383] Lustre: Permanently reactivating lustre-OST0000 [ 5464.780653] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 5464.782102] Lustre: Skipped 2 previous similar messages [ 5494.751147] LustreError: 430403:0:(osp_dev.c:684:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5496.935976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5497.064194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5497.066888] Lustre: Skipped 1 previous similar message [ 5498.362832] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5500.669877] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5509.603435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5509.604643] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5509.605437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5509.605443] Lustre: Skipped 3 previous similar messages [ 5509.607684] Lustre: Skipped 1 previous similar message [ 5513.055838] Lustre: server umount lustre-OST0000 complete [ 5513.056999] Lustre: Skipped 1 previous similar message [ 5521.560133] LustreError: 428311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327288 with bad export cookie 8765223852101035831 [ 5521.563463] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5525.819898] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5526.648199] Key type lgssc unregistered [ 5526.747348] LNet: 433250:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5526.749552] LNet: Removed LNI 192.168.204.123@tcp [ 5527.010828] Key type .llcrypt unregistered [ 5527.011741] Key type ._llcrypt unregistered [ 5532.373240] alg: No test for adler32 (adler32-zlib) [ 5533.125172] Key type ._llcrypt registered [ 5533.126221] Key type .llcrypt registered [ 5533.160153] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 5538.528394] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5538.840125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5539.078786] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 5539.152866] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5539.154185] LNet: Accept secure, port 988 [ 5540.727076] Key type lgssc registered [ 5541.099797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5543.963659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5546.124551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5548.446091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5550.647890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5552.769625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5552.791489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.888437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5553.897855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5553.933564] Lustre: lustre-MDT0000: new disk, initializing [ 5553.956221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.961619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5555.161579] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5558.867578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5558.892333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5558.921865] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5558.938172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5558.940469] Lustre: Skipped 1 previous similar message [ 5558.974920] Lustre: lustre-MDT0001: new disk, initializing [ 5559.008076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5559.018578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5559.021742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5560.299452] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5563.816137] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5565.463814] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5567.471047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5567.498733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5567.500840] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5567.595409] Lustre: lustre-OST0000: new disk, initializing [ 5567.596916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5567.621342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5568.885894] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5570.618063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5570.621251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5572.581439] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5573.943669] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5574.015035] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5575.328355] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5575.401642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5580.768928] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5580.770368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.770990] LustreError: Skipped 1 previous similar message [ 5580.776209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5580.778868] Lustre: Skipped 1 previous similar message [ 5582.120653] Lustre: server umount lustre-OST0000 complete [ 5584.863737] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5584.865790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5584.868769] Lustre: Skipped 1 previous similar message [ 5584.870213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5585.887923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5585.888464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.440947] Lustre: server umount lustre-MDT0000 complete [ 5590.523050] LustreError: 436514:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327357 with bad export cookie 15321101389791992130 [ 5590.525404] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5590.525871] LustreError: 436514:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5590.657704] Lustre: server umount lustre-MDT0001 complete [ 5592.989354] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5593.453557] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:36:00 (1761327360) [ 5595.442451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5595.595912] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5595.616258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5596.910519] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5599.384188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5600.681775] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5602.812054] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5604.482148] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5606.435861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5606.541520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5606.543031] Lustre: Skipped 1 previous similar message [ 5607.711917] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5609.820680] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5610.400134] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5611.823832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5620.193925] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5620.195844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5620.198931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5620.200248] Lustre: Skipped 2 previous similar messages [ 5624.233410] Lustre: server umount lustre-OST0000 complete [ 5630.432385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5630.432412] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5630.432864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5630.436917] Lustre: Skipped 2 previous similar messages [ 5631.487609] Lustre: server umount lustre-MDT0000 complete [ 5632.586893] LustreError: 441104:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327399 with bad export cookie 15321101389791992970 [ 5632.589554] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5632.590134] LustreError: 441104:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5636.839598] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5637.865891] Key type lgssc unregistered [ 5637.969424] LNet: 444965:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5637.971848] LNet: Removed LNI 192.168.204.123@tcp [ 5638.234200] Key type .llcrypt unregistered [ 5638.235307] Key type ._llcrypt unregistered [ 5646.406824] alg: No test for adler32 (adler32-zlib) [ 5647.158309] Key type ._llcrypt registered [ 5647.159202] Key type .llcrypt registered [ 5647.196102] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5647.302777] Key type .llcrypt unregistered [ 5647.304326] Key type ._llcrypt unregistered [ 5653.298987] alg: No test for adler32 (adler32-zlib) [ 5654.050402] Key type ._llcrypt registered [ 5654.051308] Key type .llcrypt registered [ 5654.089133] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5654.467276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5654.742946] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 5654.822634] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5654.824073] LNet: Accept secure, port 988 [ 5656.399072] Key type lgssc registered [ 5656.793806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5659.140071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.291609] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5660.304921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5661.463095] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5663.875267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5663.981362] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5663.987153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.993972] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5664.996886] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5664.997904] LustreError: Skipped 2 previous similar messages [ 5665.194983] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5667.300623] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5669.221579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5669.326275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5670.541062] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5679.585612] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5679.586051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5679.587303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5679.591407] Lustre: Skipped 1 previous similar message [ 5683.488861] Lustre: server umount lustre-OST0000 complete [ 5684.704249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5684.704405] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5684.704982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5684.704988] Lustre: Skipped 1 previous similar message [ 5684.706982] Lustre: Skipped 1 previous similar message [ 5689.823962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5689.825531] Lustre: Skipped 2 previous similar messages [ 5690.695072] Lustre: server umount lustre-MDT0000 complete [ 5691.852141] LustreError: 447271:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327458 with bad export cookie 7311290025675612144 [ 5691.855636] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5691.991688] Lustre: server umount lustre-MDT0001 complete [ 5696.128962] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5697.024904] Key type lgssc unregistered [ 5697.134299] LNet: 450552:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5697.136552] LNet: Removed LNI 192.168.204.123@tcp [ 5697.405950] Key type .llcrypt unregistered [ 5697.406799] Key type ._llcrypt unregistered [ 5705.405741] alg: No test for adler32 (adler32-zlib) [ 5706.156311] Key type ._llcrypt registered [ 5706.157175] Key type .llcrypt registered [ 5706.188170] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5706.525190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5706.682110] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 5706.745389] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5706.746662] LNet: Accept secure, port 988 [ 5708.319089] Key type lgssc registered [ 5708.642827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5710.934785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5728.415240] LustreError: 451985:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.123@tcp: failed processing log, type 1: rc = -5 [ 5759.135186] LustreError: 451985:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.123@tcp: failed processing log, type 4: rc = -110 [ 5789.855171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5791.127943] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5793.353362] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5796.208520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.525476] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.552623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.578154] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 5813.409612] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5816.802067] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.532358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.691455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5819.705367] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5823.424374] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5826.232596] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5832.160309] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5832.164065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5832.171112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5833.188286] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5833.188863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5833.192802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5835.206575] Lustre: server umount lustre-OST0000 complete [ 5838.304167] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5838.304682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5838.306370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5838.308720] Lustre: Skipped 1 previous similar message [ 5838.309947] Lustre: Skipped 1 previous similar message [ 5843.165922] Lustre: server umount lustre-MDT0000 complete [ 5843.423991] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.427856] LustreError: Skipped 1 previous similar message [ 5844.801687] LustreError: 451993:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327611 with bad export cookie 473561557041519851 [ 5844.808664] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5844.811053] LustreError: 451993:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5844.965637] Lustre: server umount lustre-MDT0001 complete [ 5850.018059] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5850.838518] Key type lgssc unregistered [ 5850.935316] LNet: 455680:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5850.937451] LNet: Removed LNI 192.168.204.123@tcp [ 5851.203573] Key type .llcrypt unregistered [ 5851.204490] Key type ._llcrypt unregistered [ 5859.599615] alg: No test for adler32 (adler32-zlib) [ 5860.350276] Key type ._llcrypt registered [ 5860.351512] Key type .llcrypt registered [ 5860.381687] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5860.655458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5860.870277] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 5860.934616] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5860.935937] LNet: Accept secure, port 988 [ 5862.503067] Key type lgssc registered [ 5862.778857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5864.970818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5866.119870] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5866.132110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5867.327252] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5869.938576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5870.084563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5871.799517] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5875.112872] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5877.507834] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5879.463189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5879.570839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5881.460038] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5881.641791] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 5883.936262] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5890.415621] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 5890.528170] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5890.534382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.042729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5891.043692] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.046085] Lustre: Skipped 2 previous similar messages [ 5891.052648] Lustre: Skipped 1 previous similar message [ 5892.064079] Lustre: server umount lustre-MDT0000 complete [ 5892.863258] LustreError: 457120:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.23@tcp arrived at 1761327659 with bad export cookie 1481717480748620779 [ 5892.866801] LustreError: 457120:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5893.438960] LustreError: 457120:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327660 with bad export cookie 1481717480748619904 [ 5893.441996] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5896.161302] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5896.161677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5896.161908] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5896.182907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5896.189608] Lustre: Skipped 1 previous similar message [ 5898.207893] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5898.208028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5898.210804] LustreError: Skipped 2 previous similar messages [ 5899.606162] Lustre: server umount lustre-OST0000 complete [ 5902.683974] Lustre: server umount lustre-MDT0001 complete [ 5907.463222] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5908.357324] Key type lgssc unregistered [ 5908.449291] LNet: 460943:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5908.452247] LNet: Removed LNI 192.168.204.123@tcp [ 5908.686909] Key type .llcrypt unregistered [ 5908.688173] Key type ._llcrypt unregistered [ 5918.417898] alg: No test for adler32 (adler32-zlib) [ 5919.169272] Key type ._llcrypt registered [ 5919.170126] Key type .llcrypt registered [ 5919.200914] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5919.481443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5919.703480] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 5919.778261] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5919.779775] LNet: Accept secure, port 988 [ 5921.351073] Key type lgssc registered [ 5921.707578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5924.046159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5925.199796] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5925.215477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5927.172698] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5929.857610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.042666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5931.830968] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5934.290179] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5936.942382] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5939.094868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5939.200238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5941.099387] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 5941.289974] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [ 5943.857239] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5946.848439] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5946.854523] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.863171] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5950.306203] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 5950.435546] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5952.153191] Lustre: server umount lustre-MDT0001 complete [ 5955.552879] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5955.554440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5955.568761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5955.573437] Lustre: Skipped 2 previous similar messages [ 5960.008986] Lustre: server umount lustre-OST0000 complete [ 5960.673638] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5967.327174] Lustre: 465450:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761327728/real 1761327728] req@00000000dfb08c29 x1846885939687040/t0(0) o251->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761327734 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5967.440101] Lustre: server umount lustre-MDT0000 complete [ 5972.670497] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 5973.530889] Key type lgssc unregistered [ 5973.635338] LNet: 466205:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5973.637405] LNet: Removed LNI 192.168.204.123@tcp [ 5973.881168] Key type .llcrypt unregistered [ 5973.882265] Key type ._llcrypt unregistered [ 5984.958718] alg: No test for adler32 (adler32-zlib) [ 5985.711297] Key type ._llcrypt registered [ 5985.712210] Key type .llcrypt registered [ 5985.753347] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 5992.789230] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 5993.139524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5993.305399] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 5993.373928] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 5993.375884] LNet: Accept secure, port 988 [ 5994.951096] Key type lgssc registered [ 5995.382500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5999.484411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6001.946748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6003.643608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6005.398894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6007.947977] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6010.434993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6012.296529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6012.315790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6013.400668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6013.409150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6013.447696] Lustre: lustre-MDT0000: new disk, initializing [ 6013.469837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6013.475177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6014.681749] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 6017.871287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6019.779753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6019.795720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6019.797723] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6019.889183] Lustre: lustre-OST0000: new disk, initializing [ 6019.890675] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6019.892139] Lustre: Skipped 1 previous similar message [ 6019.909152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.070922] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 6024.401153] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6027.370415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6029.346232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6029.367666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6029.395805] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6029.411584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6029.445956] Lustre: lustre-MDT0001: new disk, initializing [ 6029.462606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6029.464862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6029.479572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6030.706820] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 6034.029519] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 6034.092555] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6035.279433] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 6036.359967] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 6037.865880] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 6041.057222] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6041.057589] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6041.058105] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6041.062592] Lustre: Skipped 1 previous similar message [ 6042.977089] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 6042.978954] Lustre: Skipped 1 previous similar message [ 6046.176677] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6046.178150] Lustre: Skipped 1 previous similar message [ 6051.296805] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6051.298370] Lustre: Skipped 2 previous similar messages [ 6052.319124] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6052.384604] Lustre: server umount lustre-MDT0001 complete [ 6053.217583] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6056.416794] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6056.420174] LustreError: Skipped 1 previous similar message [ 6061.537078] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.540864] LustreError: Skipped 1 previous similar message [ 6066.656361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6066.656437] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6066.659540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6070.767206] Lustre: server umount lustre-OST0000 complete [ 6071.777721] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6084.575146] Lustre: 474626:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761327845/real 1761327845] req@000000009c1415dd x1846886017285952/t0(0) o251->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761327851 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6084.637298] Lustre: server umount lustre-MDT0000 complete [ 6089.874833] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6090.888433] Key type lgssc unregistered [ 6091.000309] LNet: 475381:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6091.003492] LNet: Removed LNI 192.168.204.123@tcp [ 6091.325115] Key type .llcrypt unregistered [ 6091.327267] Key type ._llcrypt unregistered [ 6104.079627] alg: No test for adler32 (adler32-zlib) [ 6104.832293] Key type ._llcrypt registered [ 6104.833243] Key type .llcrypt registered [ 6104.890246] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6105.027477] Key type .llcrypt unregistered [ 6105.029420] Key type ._llcrypt unregistered [ 6111.386765] alg: No test for adler32 (adler32-zlib) [ 6112.139892] Key type ._llcrypt registered [ 6112.140977] Key type .llcrypt registered [ 6112.200663] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6112.624562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6112.927748] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 6113.004652] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 6113.006269] LNet: Accept secure, port 988 [ 6114.591153] Key type lgssc registered [ 6115.126495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6118.098305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6120.402912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6122.595558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6125.213772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6125.254218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.386391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6126.396343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6126.439614] Lustre: lustre-MDT0000: new disk, initializing [ 6126.466425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6126.474880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6128.087432] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 6132.270803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6132.298338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6132.329818] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6132.346976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6132.348651] Lustre: Skipped 1 previous similar message [ 6132.389827] Lustre: lustre-MDT0001: new disk, initializing [ 6132.421191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6132.433247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6132.435704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6133.783021] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 6138.029264] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6140.148589] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6142.872696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6142.918842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6142.924415] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6143.065752] Lustre: lustre-OST0000: new disk, initializing [ 6143.067836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6143.105559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6144.662682] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 6149.260240] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6152.182660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6152.186734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6157.281184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.282489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6157.287490] Lustre: Skipped 2 previous similar messages [ 6157.288837] Lustre: Skipped 1 previous similar message [ 6162.400391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6162.402021] Lustre: Skipped 3 previous similar messages [ 6162.785623] Lustre: server umount lustre-MDT0000 complete [ 6164.363620] LustreError: 480956:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327931 with bad export cookie 9342336235782379958 [ 6164.368242] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6164.369858] LustreError: 480956:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6164.546333] Lustre: server umount lustre-MDT0001 complete [ 6172.639175] Lustre: 482125:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761327933/real 1761327933] req@00000000012ee0f8 x1846886143113280/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761327939 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6172.779640] Lustre: server umount lustre-OST0000 complete [ 6176.694477] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 13:45:43 (1761327943) [ 6177.140959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6179.857381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.064600] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6180.097792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.806588] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 6184.926625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6185.062164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6186.749834] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 6189.418030] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6191.360837] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6193.604177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6193.641850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6193.713328] Lustre: Found index 0 for lustre-OST0000, updating log [ 6193.769069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6195.087935] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 6196.840156] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [ 6197.399996] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6200.928468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6200.928866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.641930] Lustre: server umount lustre-OST0000 complete [ 6206.432160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6206.432740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6206.435368] Lustre: Skipped 1 previous similar message [ 6206.437922] Lustre: Skipped 2 previous similar messages [ 6211.296549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6211.298189] Lustre: Skipped 1 previous similar message [ 6213.389867] Lustre: server umount lustre-MDT0000 complete [ 6214.582970] LustreError: 482932:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327981 with bad export cookie 9342336235782381281 [ 6214.585719] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6214.587244] LustreError: 482932:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6217.664556] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 13:46:24 (1761327984) [ 6219.598888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6221.086218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6222.664272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6224.115994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6228.998123] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 6231.889450] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6235.050050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6237.278492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6239.263470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6241.279508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6243.390749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6243.414821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6243.492266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6243.500903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6243.536329] Lustre: lustre-MDT0000: new disk, initializing [ 6243.557207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6243.562558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6244.927205] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 6248.803952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6248.824077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6248.855328] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6248.870118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6248.871764] Lustre: Skipped 1 previous similar message [ 6248.907229] Lustre: lustre-MDT0001: new disk, initializing [ 6248.943735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6248.947242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6250.272369] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 6253.996157] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6255.815683] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6257.958049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6257.979135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6257.981120] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6258.071793] Lustre: lustre-OST0000: new disk, initializing [ 6258.073570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6259.759597] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug -1 all 8 [ 6264.115985] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6265.380576] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6267.379901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6267.384426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6267.491235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6268.763834] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6268.825655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6272.481791] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6272.482137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6272.482663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6272.483807] LustreError: Skipped 1 previous similar message [ 6272.487500] Lustre: Skipped 1 previous similar message [ 6275.501393] Lustre: server umount lustre-OST0000 complete [ 6275.502554] Lustre: Skipped 1 previous similar message [ 6277.600263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6277.603554] Lustre: Skipped 1 previous similar message [ 6282.719832] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6282.722896] LustreError: Skipped 1 previous similar message [ 6283.775132] LustreError: 492050:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761328050 with bad export cookie 9342336235782382324 [ 6283.777830] LustreError: 166-1: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6283.779527] LustreError: 492050:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6290.526257] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 6292.923523] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 6296.100582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6298.330518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6300.124979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6302.007234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6302.476041] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6103 sec ========== 13:47:49 (1761328069) [ 6332.083831] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 6332.908036] Key type lgssc unregistered [ 6333.011274] LNet: 498722:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6333.013598] LNet: Removed LNI 192.168.204.123@tcp [ 6333.256808] Key type .llcrypt unregistered [ 6333.257845] Key type ._llcrypt unregistered