[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 533451081 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003312] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005000] kvm-guest: setup PV IPIs [ 0.006771] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008008] pid_max: default: 32768 minimum: 301 [ 0.009113] LSM: Security Framework initializing [ 0.010032] Yama: becoming mindful. [ 0.011028] SELinux: Initializing. [ 0.012048] *** VALIDATE selinux *** [ 0.021165] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025756] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027049] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028100] *** VALIDATE tmpfs *** [ 0.029415] *** VALIDATE proc *** [ 0.030176] *** VALIDATE cgroup *** [ 0.031016] *** VALIDATE cgroup2 *** [ 0.033209] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035071] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037045] Spectre V2 : User space: Vulnerable [ 0.038012] Speculative Store Bypass: Vulnerable [ 0.040000] debug: unmapping init [mem 0xffffffffad059000-0xffffffffad060fff] [ 0.042208] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044034] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045031] ... version: 2 [ 0.046016] ... bit width: 48 [ 0.047015] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050027] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052383] rcu: Hierarchical SRCU implementation. [ 0.054646] smp: Bringing up secondary CPUs ... [ 0.055570] x86: Booting SMP configuration: [ 0.056017] .... node #0, CPUs: #1 #2 #3 [ 0.091073] smp: Brought up 1 node, 4 CPUs [ 0.093013] smpboot: Max logical packages: 1 [ 0.094021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.212147] node 0 deferred pages initialised in 115ms [ 0.216187] devtmpfs: initialized [ 0.217362] x86/mm: Memory block size: 128MB [ 0.221274] gcov: version magic: 0x41383552 [ 0.225429] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.230115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.233290] pinctrl core: initialized pinctrl subsystem [ 0.234161] [ 0.234647] ************************************************************* [ 0.237013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.239011] ** ** [ 0.242022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.244014] ** ** [ 0.245010] ** This means that this kernel is built to expose internal ** [ 0.247012] ** IOMMU data structures, which may compromise security on ** [ 0.249011] ** your system. ** [ 0.251011] ** ** [ 0.252011] ** If you see this message and you are not debugging the ** [ 0.254013] ** kernel, report this immediately to your vendor! ** [ 0.256012] ** ** [ 0.258019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.261014] ************************************************************* [ 0.262829] NET: Registered protocol family 16 [ 0.264452] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.267066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.270072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.272458] cpuidle: using governor menu [ 0.274000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.275764] PCI: Using configuration type 1 for base access [ 0.278288] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.288244] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.290013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.293118] cryptd: max_cpu_qlen set to 1000 [ 0.297170] ACPI: Added _OSI(Module Device) [ 0.300014] ACPI: Added _OSI(Processor Device) [ 0.301018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.302008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.306901] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.313269] ACPI: Interpreter enabled [ 0.314061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.315008] ACPI: Using IOAPIC for interrupt routing [ 0.316090] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.319353] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.327854] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.329022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.331014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.333060] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.337170] acpiphp: Slot [2] registered [ 0.338128] acpiphp: Slot [3] registered [ 0.339082] acpiphp: Slot [4] registered [ 0.340061] acpiphp: Slot [5] registered [ 0.341000] acpiphp: Slot [6] registered [ 0.341000] acpiphp: Slot [7] registered [ 0.342099] acpiphp: Slot [8] registered [ 0.344066] acpiphp: Slot [9] registered [ 0.346090] acpiphp: Slot [10] registered [ 0.347068] acpiphp: Slot [11] registered [ 0.348063] acpiphp: Slot [12] registered [ 0.349082] acpiphp: Slot [13] registered [ 0.350089] acpiphp: Slot [14] registered [ 0.352066] acpiphp: Slot [15] registered [ 0.353000] acpiphp: Slot [16] registered [ 0.353000] acpiphp: Slot [17] registered [ 0.355080] acpiphp: Slot [18] registered [ 0.356072] acpiphp: Slot [19] registered [ 0.358108] acpiphp: Slot [20] registered [ 0.360078] acpiphp: Slot [21] registered [ 0.361067] acpiphp: Slot [22] registered [ 0.362066] acpiphp: Slot [23] registered [ 0.368131] acpiphp: Slot [24] registered [ 0.369082] acpiphp: Slot [25] registered [ 0.370084] acpiphp: Slot [26] registered [ 0.375296] acpiphp: Slot [27] registered [ 0.377206] acpiphp: Slot [28] registered [ 0.379124] acpiphp: Slot [29] registered [ 0.380000] acpiphp: Slot [30] registered [ 0.380000] acpiphp: Slot [31] registered [ 0.381075] PCI host bridge to bus 0000:00 [ 0.383019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.385031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.388020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.390018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.393025] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.395027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.397165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.401900] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.406824] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.422019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.430054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.434034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.441019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.443016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.446444] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.448758] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.451036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.455238] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.464034] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.485030] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.489010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.490000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.497017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.502017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.515017] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.531000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.536020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.545017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.558016] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.569829] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.582031] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.587000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.602026] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.610737] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.617015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.624016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.637015] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.644489] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.655012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.663000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.673012] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.682083] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.686014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.690015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.702023] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.712000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.713334] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.715306] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.718305] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.719164] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.724092] iommu: Default domain type: Passthrough [ 0.725338] SCSI subsystem initialized [ 0.727131] ACPI: bus type USB registered [ 0.728083] usbcore: registered new interface driver usbfs [ 0.730145] usbcore: registered new interface driver hub [ 0.731061] usbcore: registered new device driver usb [ 0.733124] pps_core: LinuxPPS API ver. 1 registered [ 0.734000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.735065] PTP clock support registered [ 0.737134] EDAC MC: Ver: 3.0.0 [ 0.739161] PCI: Using ACPI for IRQ routing [ 0.740000] NetLabel: Initializing [ 0.740000] NetLabel: domain hash size = 128 [ 0.743020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.746076] NetLabel: unlabeled traffic allowed by default [ 0.748133] vgaarb: loaded [ 0.750364] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.753011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.760040] clocksource: Switched to clocksource kvm-clock [ 0.874314] VFS: Disk quotas dquot_6.6.0 [ 0.875669] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.878302] *** VALIDATE ramfs *** [ 0.879409] *** VALIDATE hugetlbfs *** [ 0.880610] pnp: PnP ACPI init [ 0.882613] pnp: PnP ACPI: found 6 devices [ 0.903517] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.912055] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.914390] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.917245] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.921851] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.924413] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.927240] NET: Registered protocol family 2 [ 0.929825] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.934615] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.938491] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.945723] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.950479] TCP: Hash tables configured (established 65536 bind 65536) [ 0.953984] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.957300] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960359] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.962799] NET: Registered protocol family 1 [ 0.965772] RPC: Registered named UNIX socket transport module. [ 0.969555] RPC: Registered udp transport module. [ 0.970826] RPC: Registered tcp transport module. [ 0.976952] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.982048] NET: Registered protocol family 44 [ 0.984069] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.988145] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.990039] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.992784] PCI: CLS 0 bytes, default 64 [ 0.994688] Unpacking initramfs... [ 3.004155] debug: unmapping init [mem 0xffff9fbb7cc54000-0xffff9fbb7ffbffff] [ 3.011908] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.019192] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.028363] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.843183] Initialise system trusted keyrings [ 3.848661] Key type blacklist registered [ 3.854084] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.870991] zbud: loaded [ 3.878625] *** VALIDATE nfs *** [ 3.879620] *** VALIDATE nfs4 *** [ 3.887379] pstore: using deflate compression [ 3.890537] Platform Keyring initialized [ 4.086473] NET: Registered protocol family 38 [ 4.087828] Key type asymmetric registered [ 4.088838] Asymmetric key parser 'x509' registered [ 4.090094] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.096751] io scheduler mq-deadline registered [ 4.097899] io scheduler kyber registered [ 4.098961] io scheduler bfq registered [ 4.100990] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.103258] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.107901] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.110767] ACPI: Power Button [PWRF] [ 4.292253] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.466329] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.915133] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.117772] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.521119] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.567989] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.612037] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.619671] Non-volatile memory driver v1.3 [ 5.620995] Linux agpgart interface v0.103 [ 5.673322] virtio_blk virtio1: [vda] 67976 512-byte logical blocks (34.8 MB/33.2 MiB) [ 5.678845] vda: detected capacity change from 0 to 34803712 [ 5.701098] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.708727] vdb: detected capacity change from 0 to 1073741824 [ 5.727894] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.730528] vdc: detected capacity change from 0 to 2621440000 [ 5.752173] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.757980] vdd: detected capacity change from 0 to 2621440000 [ 5.783252] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.792441] vde: detected capacity change from 0 to 4294967296 [ 5.819535] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.823572] vdf: detected capacity change from 0 to 4294967296 [ 5.833424] libphy: Fixed MDIO Bus: probed [ 5.840887] usbcore: registered new interface driver usbserial_generic [ 5.843389] usbserial: USB Serial support registered for generic [ 5.845115] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.849111] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.851033] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.854170] mousedev: PS/2 mouse device common for all mice [ 5.858342] rtc_cmos 00:05: RTC can wake from S4 [ 5.860337] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.864603] rtc_cmos 00:05: registered as rtc0 [ 5.869634] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.884463] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.884504] intel_pstate: CPU model not supported [ 5.902827] hid: raw HID events driver (C) Jiri Kosina [ 5.922966] usbcore: registered new interface driver usbhid [ 5.925075] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.929843] usbhid: USB HID core driver [ 5.931637] drop_monitor: Initializing network drop monitor service [ 5.933986] Initializing XFRM netlink socket [ 5.936284] NET: Registered protocol family 10 [ 5.940448] Segment Routing with IPv6 [ 5.941656] NET: Registered protocol family 17 [ 5.943236] mpls_gso: MPLS GSO support [ 5.950030] RAS: Correctable Errors collector initialized. [ 5.952333] AVX version of gcm_enc/dec engaged. [ 5.954250] AES CTR mode by8 optimization enabled [ 6.066458] sched_clock: Marking stable (6066041279, 0)->(7075890572, -1009849293) [ 6.069134] registered taskstats version 1 [ 6.070418] Loading compiled-in X.509 certificates [ 6.072498] zswap: loaded using pool lzo/zbud [ 6.105443] Key type big_key registered [ 6.139515] Key type encrypted registered [ 6.140774] ima: No TPM chip found, activating TPM-bypass! [ 6.142435] ima: Allocated hash algorithm: sha1 [ 6.143823] ima: No architecture policies found [ 6.145165] evm: Initialising EVM extended attributes: [ 6.146992] evm: security.selinux [ 6.148130] evm: security.ima [ 6.149121] evm: security.capability [ 6.150311] evm: HMAC attrs: 0x1 [ 6.154240] rtc_cmos 00:05: setting system clock to 2025-10-24 12:49:53 UTC (1761310193) [ 6.164262] debug: unmapping init [mem 0xffffffffae003000-0xffffffffae1fffff] [ 6.167124] debug: unmapping init [mem 0xffffffffacd82000-0xffffffffad058fff] [ 6.180104] Write protecting the kernel read-only data: 28672k [ 6.186109] debug: unmapping init [mem 0xffffffffab403000-0xffffffffab5fffff] [ 6.193234] debug: unmapping init [mem 0xffffffffabd14000-0xffffffffabdfffff] [ 6.285616] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.295604] systemd[1]: Detected virtualization kvm. [ 6.297106] systemd[1]: Detected architecture x86-64. [ 6.298517] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.350288] systemd[1]: No hostname configured. [ 6.352448] systemd[1]: Set hostname to . [ 6.354870] random: systemd: uninitialized urandom read (16 bytes read) [ 6.364519] systemd[1]: Initializing machine ID from random generator. [ 6.530373] random: systemd: uninitialized urandom read (16 bytes read) [ 6.532573] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.538959] random: systemd: uninitialized urandom read (16 bytes read) [ 6.541463] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.551267] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.594538] device-mapper: uevent: version 1.0.3 [ 7.596765] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK [ 8.767085] random: fast init done ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.772961] virtio_net virtio0 ens2: renamed from eth0 [ 8.857041] scsi host0: ata_piix [ 8.897109] scsi host1: ata_piix [ 8.898312] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.900603] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.671464] random: crng init done [ 13.675735] random: 7 urandom warning(s) missed due to ratelimiting [ 14.212567] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.642501] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.086701] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.721939] SELinux: Disabled at runtime. [ 17.822379] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.837239] systemd[1]: Detected virtualization kvm. [ 17.838907] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.925729] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.934092] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.942398] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.951229] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.965602] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.971772] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.988020] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ 19.287465] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 20.072613] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.688583] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.791448] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.005313] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.059946] EDAC sbridge: Ver: 1.1.2 [ 24.306912] Key type dns_resolver registered [ 24.760684] NFS: Registering the id_resolver key type [ 24.762489] Key type id_resolver registered [ 24.764804] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg353-server login: [ 44.329777] libcfs: loading out-of-tree module taints kernel. [ 44.342319] alg: No test for adler32 (adler32-zlib) [ 45.093323] Key type ._llcrypt registered [ 45.094285] Key type .llcrypt registered [ 45.125904] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 51.426360] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 51.987053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.208950] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 52.458227] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 52.460216] LNet: Accept secure, port 988 [ 54.063103] Key type lgssc registered [ 54.584478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 60.699270] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 61.647640] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 64.823130] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 67.275435] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 69.875097] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 74.555037] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 78.573829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 78.600394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 79.679070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 79.688820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 79.722635] Lustre: lustre-MDT0000: new disk, initializing [ 79.742820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 79.748481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 80.973632] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 86.193116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.220117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 86.253030] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 86.261643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 86.263331] Lustre: Skipped 1 previous similar message [ 86.302371] Lustre: lustre-MDT0001: new disk, initializing [ 86.322332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 86.330260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 86.332907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 87.802787] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 92.939741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.966121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 92.969811] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 93.058228] Lustre: lustre-OST0000: new disk, initializing [ 93.061148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 93.078861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 94.466479] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 96.490494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 96.493100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 99.566292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.592133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 99.596810] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 99.631664] Lustre: lustre-OST0001: new disk, initializing [ 99.633280] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 99.649132] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.021575] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 102.889855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 102.892906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 106.562610] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 111.086320] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 117.402645] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing check_logdir /tmp/testlogs/ [ 118.996179] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing yml_node [ 120.325296] Lustre: DEBUG MARKER: Client: 2.15.7.9 [ 121.117784] Lustre: DEBUG MARKER: MDS: 2.15.7.9 [ 121.901909] Lustre: DEBUG MARKER: OSS: 2.15.7.9 [ 122.454506] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 08:51:50 EDT 2025 [ 124.997551] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 125.480814] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 137.695539] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 137.698776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 137.703616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 138.719941] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 138.720128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 138.724605] Lustre: Skipped 2 previous similar messages [ 138.725910] Lustre: Skipped 2 previous similar messages [ 143.298339] Lustre: server umount lustre-MDT0000 complete [ 143.839627] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 143.844970] LustreError: Skipped 3 previous similar messages [ 144.605807] LustreError: 6220:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310331 with bad export cookie 6616545787257991042 [ 144.607923] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 144.610330] LustreError: 6220:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 148.959615] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 148.959624] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 148.959917] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 148.965110] LustreError: Skipped 1 previous similar message [ 148.969295] Lustre: Skipped 2 previous similar messages [ 149.983994] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 149.991562] LustreError: Skipped 1 previous similar message [ 150.765061] Lustre: server umount lustre-MDT0001 complete [ 166.367328] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 166.405056] Lustre: server umount lustre-OST0000 complete [ 175.464600] Lustre: server umount lustre-OST0001 complete [ 180.569782] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 188.356516] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 197.982945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 204.252267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.330497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 212.167039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 216.030227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 216.077507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.222182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 216.244140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 216.289492] Lustre: lustre-MDT0000: new disk, initializing [ 216.339487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.348218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 218.659738] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 224.625938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 224.656843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.701964] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 224.729184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 224.732103] Lustre: Skipped 1 previous similar message [ 224.774873] Lustre: lustre-MDT0001: new disk, initializing [ 224.822809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 224.843172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 224.852928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 227.013238] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 232.129543] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 235.105240] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 238.386719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 238.424859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.430179] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 239.937912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.675420] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 246.113631] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 248.042376] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 248.148478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 250.142678] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 250.249971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 255.458855] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 255.459184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.464153] LustreError: Skipped 1 previous similar message [ 255.469529] Lustre: Skipped 1 previous similar message [ 255.474109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 255.482163] Lustre: Skipped 3 previous similar messages [ 257.390396] Lustre: server umount lustre-OST0000 complete [ 260.575945] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 260.577903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 260.580474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 260.585470] Lustre: Skipped 1 previous similar message [ 265.696729] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 265.702825] LustreError: Skipped 1 previous similar message [ 267.178696] LustreError: 14307:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310454 with bad export cookie 6616545787257992477 [ 267.179827] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 267.187433] LustreError: 14307:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 267.326284] Lustre: server umount lustre-MDT0001 complete [ 267.328426] Lustre: Skipped 1 previous similar message [ 269.344447] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:54:16 (1761310456) [ 270.438468] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 271.120277] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:54:18 (1761310458) [ 273.750532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 273.925803] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 273.951784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 273.954390] Lustre: Skipped 1 previous similar message [ 275.639493] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 279.034446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 280.943439] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 283.864324] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 285.951807] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 288.761623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 290.654684] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 293.411806] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 314.850748] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 314.850946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.857974] LustreError: Skipped 1 previous similar message [ 314.866956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 314.869338] Lustre: Skipped 1 previous similar message [ 318.130604] Lustre: server umount lustre-OST0000 complete [ 320.481204] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 328.536579] LustreError: 18839:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310515 with bad export cookie 6616545787257993338 [ 328.537142] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 328.541686] LustreError: 18839:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 335.618497] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 337.110285] Key type lgssc unregistered [ 337.307603] LNet: 22533:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.312599] LNet: Removed LNI 192.168.203.153@tcp [ 337.788079] Key type .llcrypt unregistered [ 337.790289] Key type ._llcrypt unregistered [ 350.133122] alg: No test for adler32 (adler32-zlib) [ 350.886461] Key type ._llcrypt registered [ 350.887842] Key type .llcrypt registered [ 350.947617] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 351.443986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 351.733209] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 351.851959] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 351.859389] LNet: Accept secure, port 988 [ 353.471120] Key type lgssc registered [ 354.059764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 358.665542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.917353] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.949650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.417085] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 365.026152] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.459096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.684382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 370.137373] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 374.723407] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 378.255457] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 382.192225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 382.399048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.983461] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 389.631925] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 390.577345] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 400.865442] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 400.866390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 400.870299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 400.883039] Lustre: Skipped 1 previous similar message [ 400.894562] Lustre: Skipped 1 previous similar message [ 405.879237] Lustre: server umount lustre-OST0000 complete [ 408.546445] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 408.549073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.557976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 411.108329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 411.109670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 416.227629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 416.231611] Lustre: Skipped 2 previous similar messages [ 422.367749] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 422.368147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 422.376547] Lustre: Skipped 3 previous similar messages [ 422.434029] Lustre: server umount lustre-MDT0000 complete [ 424.366640] LustreError: 23992:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310611 with bad export cookie 12570999437594554120 [ 424.370812] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 424.371842] LustreError: 23992:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 424.535837] Lustre: server umount lustre-MDT0001 complete [ 430.724463] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 431.994193] Key type lgssc unregistered [ 432.128397] LNet: 27665:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.132644] LNet: Removed LNI 192.168.203.153@tcp [ 432.508396] Key type .llcrypt unregistered [ 432.509924] Key type ._llcrypt unregistered [ 442.952077] alg: No test for adler32 (adler32-zlib) [ 443.705481] Key type ._llcrypt registered [ 443.707069] Key type .llcrypt registered [ 443.752335] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 444.152814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 444.401576] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 444.504863] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 444.507715] LNet: Accept secure, port 988 [ 446.103132] Key type lgssc registered [ 446.596796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 450.293964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.498664] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.526904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.391848] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 456.675712] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.510946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.693395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 459.589894] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 462.966125] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 465.334114] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 468.126157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 468.277464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.940260] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 472.811674] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 482.599582] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 487.905087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.905769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 487.910376] Lustre: Skipped 2 previous similar messages [ 487.914919] Lustre: Skipped 2 previous similar messages [ 488.660473] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 492.167711] Lustre: server umount lustre-MDT0000 complete [ 493.025345] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.031030] LustreError: Skipped 2 previous similar messages [ 493.594285] LustreError: 31072:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310680 with bad export cookie 14641606243916332682 [ 493.597152] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 493.599860] LustreError: 31072:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 493.780728] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.53@tcp (stopping) [ 498.145019] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.145223] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.150531] LustreError: Skipped 1 previous similar message [ 498.159561] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 499.770866] Lustre: server umount lustre-MDT0001 complete [ 526.303153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 526.380396] Lustre: server umount lustre-OST0000 complete [ 533.999278] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 535.276673] Key type lgssc unregistered [ 535.416748] LNet: 33123:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 535.420865] LNet: Removed LNI 192.168.203.153@tcp [ 535.776655] Key type .llcrypt unregistered [ 535.778771] Key type ._llcrypt unregistered [ 546.039326] alg: No test for adler32 (adler32-zlib) [ 546.791465] Key type ._llcrypt registered [ 546.792940] Key type .llcrypt registered [ 546.842743] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 547.327727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 547.525973] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 547.617376] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 547.619678] LNet: Accept secure, port 988 [ 549.207131] Key type lgssc registered [ 549.673595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 552.915402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.110659] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.131178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.912282] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 559.409378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.576786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 561.290265] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 564.151665] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 566.348341] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 568.929398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 569.072990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 570.663705] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 573.412456] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 584.421083] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 594.132785] Lustre: lustre-OST0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 594.914800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.915034] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 594.917303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 594.920463] Lustre: Skipped 1 previous similar message [ 597.608610] Lustre: server umount lustre-OST0000 complete [ 615.904401] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 615.917100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.931203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 615.942319] Lustre: Skipped 1 previous similar message [ 618.324740] Lustre: server umount lustre-MDT0000 complete [ 621.554169] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.580766] LustreError: Skipped 1 previous similar message [ 622.238977] LustreError: 35383:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310809 with bad export cookie 10135752036201999649 [ 622.244298] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 622.262510] LustreError: 35383:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 628.795519] Lustre: server umount lustre-MDT0001 complete [ 640.985845] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 644.399230] Key type lgssc unregistered [ 644.677870] LNet: 38445:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.695987] LNet: Removed LNI 192.168.203.153@tcp [ 645.554915] Key type .llcrypt unregistered [ 645.559573] Key type ._llcrypt unregistered [ 651.441190] hrtimer: interrupt took 3958312 ns [ 667.219452] alg: No test for adler32 (adler32-zlib) [ 667.982460] Key type ._llcrypt registered [ 667.983916] Key type .llcrypt registered [ 668.089527] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 669.189069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 669.459514] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 669.623882] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 669.628212] LNet: Accept secure, port 988 [ 671.293142] Key type lgssc registered [ 672.294571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 679.582341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.048071] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 681.080402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 684.327276] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 686.563278] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.710836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.936145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 693.503763] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 698.103994] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 701.309142] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 705.499866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 705.775382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 708.610424] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 710.889729] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 713.543285] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 717.598683] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 719.798389] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 724.975193] LNetError: 120-3: Refusing connection from 192.168.203.153 for 127.0.0.2@tcp: No matching NI [ 724.981630] LNetError: 39211:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 724.989151] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 731.075062] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-24 9h01m58s [ 732.220953] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 735.454983] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 736.738428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.741150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 736.764232] Lustre: Skipped 2 previous similar messages [ 736.771813] Lustre: Skipped 1 previous similar message [ 736.784549] LNetError: 120-3: Refusing connection from 192.168.203.153 for 127.0.0.2@tcp: No matching NI [ 736.789158] LNetError: Skipped 1 previous similar message [ 736.792080] LNetError: 39209:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 736.797183] LNetError: 39209:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Skipped 1 previous similar message [ 736.801173] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 736.809424] LNetError: Skipped 1 previous similar message [ 739.768035] Lustre: server umount lustre-MDT0000 complete [ 740.562351] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.866711] LNetError: 120-3: Refusing connection from 192.168.203.153 for 127.0.0.2@tcp: No matching NI [ 741.870608] LNetError: 39210:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 741.875217] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 742.182682] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 745.679119] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 745.686941] LustreError: Skipped 3 previous similar messages [ 746.683789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 746.766106] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 746.771653] Lustre: Evicted from MGS (at 192.168.203.153@tcp) after server handle changed from 0xb5c773515d417bbb to 0xb5c773515d41801b [ 746.786966] Lustre: MGC192.168.203.153@tcp: Connection restored to (at 0@lo) [ 746.938112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.981452] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 749.483516] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 751.619206] Lustre: DEBUG MARKER: Wait for df (28867) ... [ 752.101171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 752.112522] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 752.894446] Lustre: DEBUG MARKER: done [ 757.219854] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 757.221847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 757.225958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 757.230334] Lustre: Skipped 1 previous similar message [ 757.236919] Lustre: Skipped 2 previous similar messages [ 761.456691] Lustre: server umount lustre-OST0000 complete [ 767.458129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.458406] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 767.459621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 767.465549] Lustre: Skipped 1 previous similar message [ 767.476895] LNetError: 120-3: Refusing connection from 192.168.203.153 for 127.0.0.2@tcp: No matching NI [ 767.484518] LNetError: 39211:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 767.488903] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 769.829955] Lustre: server umount lustre-MDT0000 complete [ 771.975372] LustreError: 42619:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310959 with bad export cookie 13098564834524495899 [ 771.979313] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 771.981639] LustreError: 42619:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 772.150495] Lustre: server umount lustre-MDT0001 complete [ 779.098637] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 780.927368] Key type lgssc unregistered [ 781.126668] LNet: 45018:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.133737] LNet: Removed LNI 192.168.203.153@tcp [ 781.570247] Key type .llcrypt unregistered [ 781.572352] Key type ._llcrypt unregistered [ 811.771735] alg: No test for adler32 (adler32-zlib) [ 812.526447] Key type ._llcrypt registered [ 812.528875] Key type .llcrypt registered [ 812.617793] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 824.218712] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 824.924494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 825.132866] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 825.286243] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 825.289435] LNet: Accept secure, port 988 [ 826.919122] Key type lgssc registered [ 827.637977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 833.855482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 838.743512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 843.191498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 847.915691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 852.592916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 852.641396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 853.782752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 853.810379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 853.860943] Lustre: lustre-MDT0000: new disk, initializing [ 853.940219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 853.964814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 856.582734] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 863.869507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 863.962044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 864.019879] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 864.064755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 864.068288] Lustre: Skipped 1 previous similar message [ 864.127780] Lustre: lustre-MDT0001: new disk, initializing [ 864.195132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 864.235305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 864.247077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 867.366165] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 873.549967] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 876.543929] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 880.206221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 880.253645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 880.260329] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 880.377333] Lustre: lustre-OST0000: new disk, initializing [ 880.380710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 880.421293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 882.581576] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 887.340580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 887.351392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 888.885301] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 891.525297] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 891.663535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 894.195955] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 894.324763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 897.504857] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 897.505622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.508606] LustreError: Skipped 1 previous similar message [ 897.515621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 897.520657] Lustre: Skipped 1 previous similar message [ 901.619779] Lustre: server umount lustre-OST0000 complete [ 905.185677] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 905.192673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.199456] Lustre: Skipped 1 previous similar message [ 905.202440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 907.745245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.746205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 907.761558] Lustre: Skipped 1 previous similar message [ 909.647673] Lustre: server umount lustre-MDT0000 complete [ 911.569115] LustreError: 51190:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311098 with bad export cookie 16975585505381593778 [ 911.571441] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 911.574044] LustreError: 51190:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 911.718696] Lustre: server umount lustre-MDT0001 complete [ 915.810729] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 09:05:03 (1761311103) [ 919.142173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.372250] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 919.427190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 921.557142] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 924.643530] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 925.632425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.801324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 927.638501] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 931.004191] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 933.386513] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 936.420557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 936.593638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.452742] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 941.723513] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 955.234392] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-24 9h05m42s [ 955.924715] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 958.937483] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 959.994235] Lustre: DEBUG MARKER: Creating a test file and stat it [ 960.010622] LustreError: 56728:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 964.584855] LNetError: 120-3: Refusing connection from 192.168.203.153 for 127.0.0.2@tcp: No matching NI [ 964.588213] LNetError: 47326:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 964.591315] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 980.200884] Lustre: lustre-MDT0000: Client ef82312b-7b36-4f69-bb96-24eda2e2b2c9 (at 192.168.203.53@tcp) reconnecting [ 1000.063113] LustreError: 56728:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1000.995240] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1002.128150] Lustre: DEBUG MARKER: done [ 1005.538451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.541743] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1005.542215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1005.546533] Lustre: Skipped 1 previous similar message [ 1010.423502] Lustre: server umount lustre-OST0000 complete [ 1012.704341] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1012.708065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.714823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1012.717645] Lustre: Skipped 1 previous similar message [ 1015.779532] LNetError: 120-3: Refusing connection from 192.168.203.153 for 127.0.0.2@tcp: No matching NI [ 1015.783374] LNetError: 47325:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1015.786061] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1018.507090] Lustre: server umount lustre-MDT0000 complete [ 1020.152694] LustreError: 53643:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311207 with bad export cookie 16975585505381594632 [ 1020.153445] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.159539] LustreError: 53643:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1025.822866] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1027.169157] Key type lgssc unregistered [ 1027.313688] LNet: 58092:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.316602] LNet: Removed LNI 192.168.203.153@tcp [ 1027.688752] Key type .llcrypt unregistered [ 1027.690823] Key type ._llcrypt unregistered [ 1043.766737] alg: No test for adler32 (adler32-zlib) [ 1044.521281] Key type ._llcrypt registered [ 1044.522762] Key type .llcrypt registered [ 1044.578402] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 1053.165843] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1053.798704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.174521] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 1054.288819] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1054.291304] LNet: Accept secure, port 988 [ 1055.903313] Key type lgssc registered [ 1056.545110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1061.532885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1065.120395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1068.295032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1071.282978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1074.948984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1074.983388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.094282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1076.115590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1076.166423] Lustre: lustre-MDT0000: new disk, initializing [ 1076.232413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.246958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1078.416295] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1084.141221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1084.179792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.229840] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1084.258291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1084.261458] Lustre: Skipped 1 previous similar message [ 1084.307805] Lustre: lustre-MDT0001: new disk, initializing [ 1084.363210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1084.382994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1084.387793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1086.225478] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1090.956867] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1093.634334] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1097.333784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1097.386633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1097.391743] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1097.541303] Lustre: lustre-OST0000: new disk, initializing [ 1097.543622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1097.596318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1099.958321] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1104.488695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1104.500569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1110.609391] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1116.959262] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1117.313249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1122.630119] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1123.027688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1130.465840] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1130.467747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.498662] LustreError: Skipped 1 previous similar message [ 1130.508460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1130.539833] Lustre: Skipped 1 previous similar message [ 1130.565587] Lustre: Skipped 1 previous similar message [ 1131.872333] Lustre: server umount lustre-OST0000 complete [ 1140.704150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.743435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1140.758985] Lustre: Skipped 1 previous similar message [ 1143.245563] Lustre: server umount lustre-MDT0000 complete [ 1145.825169] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1145.835195] LustreError: Skipped 1 previous similar message [ 1146.686301] LustreError: 62175:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311334 with bad export cookie 8336066478228873714 [ 1146.687136] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1146.697316] LustreError: 62175:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1147.004651] Lustre: server umount lustre-MDT0001 complete [ 1153.277982] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 09:09:00 (1761311340) [ 1154.195508] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1155.343254] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 09:09:02 (1761311342) [ 1161.802727] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1164.332086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.689657] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.743356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1165.317704] Lustre: Failing over lustre-MDT0000 [ 1165.321424] LustreError: 67195:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1165.329458] LustreError: 67195:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1165.335192] LustreError: 67195:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1170.911154] Lustre: 67246:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311352/real 1761311352] req@000000000526a2c6 x1846868695863616/t0(0) o251->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761311358 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1171.041126] Lustre: server umount lustre-MDT0000 complete [ 1175.381127] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:09:22 (1761311362) [ 1179.842164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.099242] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.134714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1182.711477] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1185.251019] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.011626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1189.286353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1192.491690] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1197.340657] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1201.136469] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1205.615819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1205.849139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.429821] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1218.289219] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1225.462587] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1237.321268] Lustre: server umount lustre-MDT0000 complete [ 1240.040561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.043888] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.081122] Lustre: Skipped 3 previous similar messages [ 1240.110991] LustreError: Skipped 2 previous similar messages [ 1243.977895] LustreError: 67793:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311431 with bad export cookie 8336066478228874827 [ 1243.992842] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1244.006577] LustreError: 67793:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1248.620600] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1252.335072] Lustre: 60409:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311432/real 1761311432] req@0000000030fd2cdd x1846868695881984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311439 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1252.346775] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.291418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1255.264993] Lustre: 60407:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311435/real 1761311435] req@000000000a8145ff x1846868695882048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311442 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1257.375477] Lustre: 60407:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311437/real 1761311437] req@000000009289b40d x1846868695882176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311444 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1263.710794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.963430] Lustre: Evicted from MGS (at 192.168.203.153@tcp) after server handle changed from 0x73afa84baadfa64b to 0x73afa84baadfadde [ 1276.976911] Lustre: MGC192.168.203.153@tcp: Connection restored to 192.168.203.153@tcp (at 0@lo) [ 1277.325938] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.400951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.725985] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1289.656910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1292.971357] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1293.232280] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1299.850526] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1304.870694] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1310.688468] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1310.701511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.731397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.934842] Lustre: server umount lustre-MDT0000 complete [ 1311.936886] Lustre: Skipped 1 previous similar message [ 1313.764341] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.780658] LustreError: Skipped 5 previous similar messages [ 1315.245326] LustreError: 67793:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311502 with bad export cookie 8336066478228876766 [ 1315.248856] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1315.261976] LustreError: 67793:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1321.649280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1327.008734] Lustre: 60408:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311506/real 1761311506] req@000000004e7ebfbf x1846868695894656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311513 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1327.028193] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1327.037551] Lustre: Skipped 2 previous similar messages [ 1329.776943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.200774] Lustre: Evicted from MGS (at 192.168.203.153@tcp) after server handle changed from 0x73afa84baadfadde to 0x73afa84baadfb356 [ 1332.222179] Lustre: MGC192.168.203.153@tcp: Connection restored to (at 0@lo) [ 1332.236639] Lustre: Skipped 1 previous similar message [ 1332.596925] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1335.926422] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1343.379505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.908300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1343.922379] Lustre: Skipped 2 previous similar messages [ 1344.017804] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1345.966772] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1348.344268] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1355.719742] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1361.651952] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1368.953126] Lustre: server umount lustre-MDT0000 complete [ 1368.956846] Lustre: Skipped 1 previous similar message [ 1369.573267] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1369.576917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.587923] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.606740] LustreError: Skipped 7 previous similar messages [ 1371.960253] LustreError: 67794:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311559 with bad export cookie 8336066478228878166 [ 1371.963754] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1371.970116] LustreError: 67794:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1377.619920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1380.498520] Lustre: DEBUG MARKER: files compared the same [ 1387.999620] Lustre: 76941:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311569/real 1761311569] req@0000000046740299 x1846868695908160/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311575 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1388.034869] Lustre: 76941:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1404.530598] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1407.952844] Key type lgssc unregistered [ 1408.185049] LNet: 77848:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1408.198624] LNet: Removed LNI 192.168.203.153@tcp [ 1409.214985] Key type .llcrypt unregistered [ 1409.217695] Key type ._llcrypt unregistered [ 1431.031989] alg: No test for adler32 (adler32-zlib) [ 1431.795904] Key type ._llcrypt registered [ 1431.805301] Key type .llcrypt registered [ 1431.995293] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1433.166775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1433.603205] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 1433.792927] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1433.797615] LNet: Accept secure, port 988 [ 1435.439794] Key type lgssc registered [ 1436.694685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1445.077675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.446844] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.504113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1449.813127] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug +malloc all 8 [ 1452.008578] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.071694] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1457.049625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1457.334249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1461.662323] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug +malloc all 8 [ 1468.622806] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1473.978808] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1479.766904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1479.985489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.693699] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug +malloc all 8 [ 1487.157572] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1490.487608] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1497.388757] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1507.809412] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1507.819831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.822065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1507.826793] LustreError: Skipped 1 previous similar message [ 1507.867019] Lustre: Skipped 1 previous similar message [ 1512.930723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1512.947358] Lustre: Skipped 1 previous similar message [ 1513.573529] Lustre: server umount lustre-OST0000 complete [ 1518.049748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.058576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1518.061281] Lustre: Skipped 2 previous similar messages [ 1523.174087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.416771] Lustre: server umount lustre-MDT0000 complete [ 1527.118972] LustreError: 79292:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311714 with bad export cookie 6304284444809508458 [ 1527.133756] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1527.431490] Lustre: server umount lustre-MDT0001 complete [ 1536.446777] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1539.145780] Key type lgssc unregistered [ 1539.407918] LNet: 82982:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.417431] LNet: Removed LNI 192.168.203.153@tcp [ 1540.147936] Key type .llcrypt unregistered [ 1540.152583] Key type ._llcrypt unregistered [ 1560.684716] alg: No test for adler32 (adler32-zlib) [ 1561.450880] Key type ._llcrypt registered [ 1561.457482] Key type .llcrypt registered [ 1561.589703] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1562.717242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1562.999184] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 1563.179935] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1563.188342] LNet: Accept secure, port 988 [ 1564.911735] Key type lgssc registered [ 1566.158672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1574.756816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1592.543318] LustreError: 84417:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.153@tcp: failed processing log, type 1: rc = -5 [ 1623.263525] LustreError: 84417:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.153@tcp: failed processing log, type 4: rc = -110 [ 1653.984336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.290439] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1663.889555] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1671.199896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.288667] Lustre: *** cfs_fail_loc=706, val=0*** [ 1675.673665] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1675.730598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.774642] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1679.674850] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1680.866643] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1685.988809] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.716324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.899561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1686.954313] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1689.783888] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1695.664059] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1699.957952] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1702.370185] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1702.386713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.405457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1703.398561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1703.400909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1707.661788] Lustre: server umount lustre-OST0000 complete [ 1712.608198] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1712.620787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.633182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1712.650167] Lustre: Skipped 1 previous similar message [ 1716.372397] Lustre: server umount lustre-MDT0000 complete [ 1718.752986] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1718.784216] LustreError: Skipped 1 previous similar message [ 1719.945238] LustreError: 84423:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311907 with bad export cookie 2266668634258481782 [ 1719.952215] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1719.957255] LustreError: 84423:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1720.279453] Lustre: server umount lustre-MDT0001 complete [ 1730.752239] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1734.044483] Key type lgssc unregistered [ 1734.400381] LNet: 88160:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.411663] LNet: Removed LNI 192.168.203.153@tcp [ 1735.677394] Key type .llcrypt unregistered [ 1735.684481] Key type ._llcrypt unregistered [ 1757.079642] alg: No test for adler32 (adler32-zlib) [ 1757.834431] Key type ._llcrypt registered [ 1757.835957] Key type .llcrypt registered [ 1757.923752] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1758.712809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.159771] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 1759.417280] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1759.420687] LNet: Accept secure, port 988 [ 1761.152360] Key type lgssc registered [ 1762.299752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1769.327636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.766711] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1779.290213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.680110] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1779.754309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1783.051952] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1784.803244] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1784.821419] LustreError: Skipped 2 previous similar messages [ 1789.161956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1789.388081] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1789.397967] LustreError: Skipped 2 previous similar messages [ 1789.421242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.009392] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1794.534589] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1794.555030] LustreError: Skipped 2 previous similar messages [ 1795.565039] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1799.325677] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1799.332227] LustreError: Skipped 5 previous similar messages [ 1799.473459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.521288] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 1802.685516] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1815.009725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.012087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1815.043752] Lustre: Skipped 1 previous similar message [ 1819.991515] Lustre: server umount lustre-OST0000 complete [ 1825.248254] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1825.253166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1825.257259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.261393] Lustre: Skipped 1 previous similar message [ 1825.267055] Lustre: Skipped 2 previous similar messages [ 1829.433901] Lustre: server umount lustre-MDT0000 complete [ 1830.368690] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.389556] LustreError: Skipped 1 previous similar message [ 1833.131539] Lustre: server umount lustre-MDT0001 complete [ 1842.655979] Lustre: 93025:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312023/real 1761312023] req@00000000de7c386e x1846869435101568/t0(0) o251->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761312029 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1842.697945] Lustre: server umount MGS complete [ 1851.047775] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:20:38 (1761312038) [ 1873.537218] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 1881.917988] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1889.707709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1894.576747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1899.405399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1904.744869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1910.404624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1910.475349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.133466] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1922.199599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1922.282978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.496536] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1922.540825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1922.619411] Lustre: lustre-MDT0001: new disk, initializing [ 1922.694442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1922.727779] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1922.731408] LustreError: 97725:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 1926.665398] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1935.907596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1935.974602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1935.977474] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1936.147703] Lustre: lustre-OST0000: new disk, initializing [ 1936.151290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1936.204366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.742956] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1945.185481] Lustre: 99016:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@0000000081ea64e2 x1846869435106304/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761312144 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1948.022143] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1953.348562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1953.389745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1953.445413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1953.460513] Lustre: lustre-MDT0000: new disk, initializing [ 1953.601080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.625894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1954.274932] LustreError: 98538:0:(fid_handler.c:334:__seq_server_alloc_meta()) srv-lustre-OST0000: Allocated super-sequence failed: rc = -11 [ 1956.330928] LustreError: 98538:0:(fid_handler.c:334:__seq_server_alloc_meta()) srv-lustre-OST0000: Allocated super-sequence failed: rc = -11 [ 1956.698639] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 1958.367813] LustreError: 98538:0:(fid_handler.c:334:__seq_server_alloc_meta()) srv-lustre-OST0000: Allocated super-sequence failed: rc = -11 [ 1958.896450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1958.904872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1960.425037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1960.429401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1979.365581] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1979.371297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.409399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1979.412106] Lustre: Skipped 2 previous similar messages [ 1980.896480] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1985.267841] Lustre: server umount lustre-OST0000 complete [ 1989.600143] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1989.607517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.617180] Lustre: Skipped 1 previous similar message [ 1989.620747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1989.626177] Lustre: Skipped 3 previous similar messages [ 1994.907346] Lustre: server umount lustre-MDT0000 complete [ 1997.793156] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1997.800309] LustreError: Skipped 3 previous similar messages [ 2014.175620] Lustre: 100770:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312195/real 1761312195] req@0000000031d856a5 x1846869435119872/t0(0) o251->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761312201 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2014.234708] Lustre: server umount MGS complete [ 2014.238054] Lustre: Skipped 1 previous similar message [ 2024.369446] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:23:31 (1761312211) [ 2039.198927] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2041.240809] Key type lgssc unregistered [ 2041.421111] LNet: 102177:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.426666] LNet: Removed LNI 192.168.203.153@tcp [ 2041.989982] Key type .llcrypt unregistered [ 2041.996133] Key type ._llcrypt unregistered [ 2054.205717] alg: No test for adler32 (adler32-zlib) [ 2054.960620] Key type ._llcrypt registered [ 2054.965938] Key type .llcrypt registered [ 2055.069808] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2055.659286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.136897] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 2056.289901] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2056.293638] LNet: Accept secure, port 988 [ 2057.927323] Key type lgssc registered [ 2058.894798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2065.143529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2066.489789] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.541545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.883542] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2072.035550] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.810359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.186817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2080.242670] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2085.991590] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2089.982160] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2095.477509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2095.773602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.818233] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2099.167063] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2105.390821] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2117.601195] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2117.614978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.635030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2120.541371] Lustre: server umount lustre-OST0000 complete [ 2123.232048] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2123.235402] LustreError: Skipped 1 previous similar message [ 2123.248424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.254512] Lustre: Skipped 1 previous similar message [ 2123.259038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.261031] Lustre: Skipped 1 previous similar message [ 2127.843218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2127.843832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.453881] Lustre: server umount lustre-MDT0000 complete [ 2132.613647] LustreError: 106004:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312319 with bad export cookie 17068772587758053994 [ 2132.617385] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.626989] LustreError: 106004:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2132.888642] Lustre: server umount lustre-MDT0001 complete [ 2141.628740] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2143.521383] Key type lgssc unregistered [ 2143.769186] LNet: 107098:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.782263] LNet: Removed LNI 192.168.203.153@tcp [ 2144.401314] Key type .llcrypt unregistered [ 2144.405974] Key type ._llcrypt unregistered [ 2155.580101] alg: No test for adler32 (adler32-zlib) [ 2156.341351] Key type ._llcrypt registered [ 2156.343479] Key type .llcrypt registered [ 2156.429274] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2157.142960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2157.781475] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 2157.993774] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2157.998098] LNet: Accept secure, port 988 [ 2159.663090] Key type lgssc registered [ 2160.672357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2166.725471] LustreError: 108224:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2166.923797] LustreError: 108242:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2166.934655] LustreError: 108224:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4788 [ 2166.937168] LustreError: 108224:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2166.962684] LustreError: 108224:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2167.030613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.991570] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.086603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.435095] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2176.642322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.995200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2180.783887] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2185.475852] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing lsmod [ 2187.873292] LustreError: 109534:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2187.994253] LustreError: 109550:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2188.007714] LustreError: 109534:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4866 [ 2188.017462] LustreError: 109534:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2188.035723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2188.056039] LustreError: 109534:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2188.399409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.353101] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2192.359652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.370139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.377276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.379314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.387250] Lustre: Skipped 1 previous similar message [ 2193.397272] Lustre: Skipped 2 previous similar messages [ 2196.868493] Lustre: server umount lustre-MDT0000 complete [ 2198.499688] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.507692] LustreError: Skipped 2 previous similar messages [ 2199.385612] LustreError: 108261:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312386 with bad export cookie 1731317080639912883 [ 2199.389083] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.390758] LustreError: 108261:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2199.577278] Lustre: server umount lustre-MDT0001 complete [ 2208.223166] Lustre: 110139:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312389/real 1761312389] req@00000000ea5c8e1e x1846869853480320/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312395 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2208.491356] Lustre: server umount lustre-OST0000 complete [ 2214.578680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.845233] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.892883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.378335] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2220.001312] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.369711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2222.585315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2225.092933] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2229.769090] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2233.224718] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2237.827971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2238.059551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2241.150738] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2245.229526] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2245.902345] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2260.960922] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2260.964405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.975721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2264.034147] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2264.034357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2264.041238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.304052] Lustre: server umount lustre-OST0000 complete [ 2269.153868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2269.159341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.161118] Lustre: Skipped 1 previous similar message [ 2269.166843] Lustre: Skipped 2 previous similar messages [ 2275.044059] Lustre: server umount lustre-MDT0000 complete [ 2277.954889] LustreError: 113484:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312465 with bad export cookie 1731317080639913583 [ 2277.963262] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.978375] LustreError: 113484:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2287.396494] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2289.522294] Key type lgssc unregistered [ 2289.737397] LNet: 114288:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2289.747377] LNet: Removed LNI 192.168.203.153@tcp [ 2290.277726] Key type .llcrypt unregistered [ 2290.282776] Key type ._llcrypt unregistered [ 2305.578353] alg: No test for adler32 (adler32-zlib) [ 2306.330345] Key type ._llcrypt registered [ 2306.332753] Key type .llcrypt registered [ 2306.481796] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2307.145519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2307.521432] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 2307.668376] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2307.673237] LNet: Accept secure, port 988 [ 2309.303257] Key type lgssc registered [ 2310.136287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2315.412651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.669113] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.714929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2320.314483] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2321.892196] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.570638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.807035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2329.741445] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2334.755183] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2338.452707] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2342.599156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2342.809237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2345.762800] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2346.926454] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2352.375597] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2359.822691] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2361.696337] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2364.915657] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2364.922674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.950151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.953767] Lustre: server umount lustre-OST0000 complete [ 2372.578585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.581420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2372.602203] Lustre: Skipped 1 previous similar message [ 2372.616935] Lustre: Skipped 1 previous similar message [ 2374.624141] Lustre: 115063:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312554/real 1761312554] req@00000000cda3bd0b x1846870010769856/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761312561 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2374.639743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.659269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2374.662160] Lustre: Skipped 1 previous similar message [ 2376.684561] Lustre: server umount lustre-MDT0000 complete [ 2378.719502] Lustre: 115062:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312559/real 1761312559] req@00000000226250cc x1846870010772160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761312566 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2378.738496] Lustre: 115062:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2379.298597] LustreError: 115731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312566 with bad export cookie 16639192406724017169 [ 2379.302863] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2379.305474] LustreError: 115731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2379.511014] Lustre: server umount lustre-MDT0001 complete [ 2388.736493] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2391.424900] Key type lgssc unregistered [ 2391.746720] LNet: 119507:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.758859] LNet: Removed LNI 192.168.203.153@tcp [ 2392.596698] Key type .llcrypt unregistered [ 2392.599825] Key type ._llcrypt unregistered [ 2408.509889] alg: No test for adler32 (adler32-zlib) [ 2409.265373] Key type ._llcrypt registered [ 2409.267504] Key type .llcrypt registered [ 2409.424688] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2410.328493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2410.710355] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 2410.865755] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2410.868939] LNet: Accept secure, port 988 [ 2412.567122] Key type lgssc registered [ 2413.621806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2419.785126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.195793] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.266519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.751921] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2426.353566] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.154918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.461828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2434.509903] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2441.347682] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2446.393065] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2452.381106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2452.590351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.229845] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2462.527901] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2471.025501] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2479.086346] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2479.093594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.101034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2481.634889] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2481.637513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2481.648879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.755669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.764262] Lustre: Skipped 2 previous similar messages [ 2491.871123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2491.878556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2491.884382] Lustre: Skipped 1 previous similar message [ 2491.982740] Lustre: server umount lustre-OST0000 complete [ 2496.996249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.998465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2497.007340] Lustre: Skipped 1 previous similar message [ 2500.722946] Lustre: server umount lustre-MDT0000 complete [ 2502.115271] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.122521] LustreError: Skipped 1 previous similar message [ 2503.420646] LustreError: 120775:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312690 with bad export cookie 13957892798324832232 [ 2503.423179] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2503.429856] LustreError: 120775:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2503.751856] Lustre: server umount lustre-MDT0001 complete [ 2512.163342] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2514.495832] Key type lgssc unregistered [ 2514.744837] LNet: 124475:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2514.754712] LNet: Removed LNI 192.168.203.153@tcp [ 2515.397903] Key type .llcrypt unregistered [ 2515.399468] Key type ._llcrypt unregistered [ 2531.866878] alg: No test for adler32 (adler32-zlib) [ 2532.618695] Key type ._llcrypt registered [ 2532.625293] Key type .llcrypt registered [ 2532.707738] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2533.457548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2533.797378] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 2533.939195] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2533.943215] LNet: Accept secure, port 988 [ 2535.591130] Key type lgssc registered [ 2536.497396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2541.992607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.261383] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.312026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.152598] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2548.706098] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.730715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.013569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2556.113377] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2562.341743] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2566.626693] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2570.245267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2570.473724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.759299] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2574.580498] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 2577.759494] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2586.049921] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2588.372257] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2588.375605] Lustre: Skipped 2 previous similar messages [ 2591.005592] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2591.010857] Lustre: Skipped 2 previous similar messages [ 2591.767697] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2591.780028] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2596.136663] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2596.139277] Lustre: Skipped 2 previous similar messages [ 2597.966240] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.53@tcp 192.168.0.[2,10]@tcp [ 2603.492443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2603.493453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2603.502188] Lustre: Skipped 1 previous similar message [ 2603.510244] Lustre: Skipped 1 previous similar message [ 2605.053559] Lustre: server umount lustre-OST0000 complete [ 2608.609063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.612261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2608.619345] Lustre: Skipped 1 previous similar message [ 2613.728785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2613.738059] Lustre: Skipped 2 previous similar messages [ 2613.985213] Lustre: server umount lustre-MDT0000 complete [ 2616.457115] LustreError: 125916:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312803 with bad export cookie 3394300770780620484 [ 2616.459968] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2616.462364] LustreError: 125916:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2616.630280] Lustre: server umount lustre-MDT0001 complete [ 2623.725110] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2625.475175] Key type lgssc unregistered [ 2625.684587] LNet: 130497:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2625.689201] LNet: Removed LNI 192.168.203.153@tcp [ 2626.270987] Key type .llcrypt unregistered [ 2626.272569] Key type ._llcrypt unregistered [ 2647.637230] alg: No test for adler32 (adler32-zlib) [ 2648.386505] Key type ._llcrypt registered [ 2648.388546] Key type .llcrypt registered [ 2648.453177] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2649.008664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2649.240353] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 2649.410371] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2649.413796] LNet: Accept secure, port 988 [ 2651.031119] Key type lgssc registered [ 2651.860928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2657.704981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.045226] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.076818] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2659.106227] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.203.53@tcp 192.168.0.[2,10]@tcp [ 2659.108376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.037517] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2664.419445] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.758730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2667.983585] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2668.002349] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.53@tcp 192.168.0.[2,10]@tcp [ 2668.020457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2670.688681] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2675.582183] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2678.736501] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2682.528648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2682.727871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.805299] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2688.882780] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 2689.379619] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2695.139283] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2695.142149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.150864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2699.233867] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2699.237390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2699.238363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.258678] Lustre: Skipped 1 previous similar message [ 2700.806725] Lustre: server umount lustre-OST0000 complete [ 2703.841411] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2703.845431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2703.854205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.474317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.478746] Lustre: Skipped 3 previous similar messages [ 2709.618684] Lustre: server umount lustre-MDT0000 complete [ 2712.160559] LustreError: 132085:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312899 with bad export cookie 2229917862260114235 [ 2712.168888] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2712.172993] LustreError: 132085:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2712.412878] Lustre: server umount lustre-MDT0001 complete [ 2720.690774] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2722.505807] Key type lgssc unregistered [ 2722.680049] LNet: 135813:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2722.686780] LNet: Removed LNI 192.168.203.153@tcp [ 2723.222550] Key type .llcrypt unregistered [ 2723.224742] Key type ._llcrypt unregistered [ 2739.561196] alg: No test for adler32 (adler32-zlib) [ 2740.315663] Key type ._llcrypt registered [ 2740.317063] Key type .llcrypt registered [ 2740.397523] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 2748.710840] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2749.284722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2749.587512] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 2749.720764] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2749.725705] LNet: Accept secure, port 988 [ 2751.354526] Key type lgssc registered [ 2752.253168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2757.761961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2761.589784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2765.791221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2770.430329] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2774.469853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2774.519580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.678810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2775.705606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2775.768754] Lustre: lustre-MDT0000: new disk, initializing [ 2775.836226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2775.849279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2778.463598] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2785.340405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2785.426775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2785.490422] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2785.518839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2785.521552] Lustre: Skipped 1 previous similar message [ 2785.569941] Lustre: lustre-MDT0001: new disk, initializing [ 2785.642785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2785.673869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2785.678537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2787.942655] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2794.265163] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2797.823812] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2802.232945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2802.287884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2802.292308] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2802.483926] Lustre: lustre-OST0000: new disk, initializing [ 2802.489866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2802.564264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2805.736418] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2809.370621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2809.377268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2812.757515] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2815.850776] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2816.012894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2819.071838] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2819.250857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2824.673305] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2824.673788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.674408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2824.676988] LustreError: Skipped 1 previous similar message [ 2824.684120] Lustre: Skipped 1 previous similar message [ 2826.626133] Lustre: server umount lustre-OST0000 complete [ 2829.796182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.798153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2829.803284] Lustre: Skipped 1 previous similar message [ 2829.813402] Lustre: Skipped 2 previous similar messages [ 2834.918389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.157275] Lustre: server umount lustre-MDT0000 complete [ 2838.164056] LustreError: 141412:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313025 with bad export cookie 7531730738018072106 [ 2838.167820] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2838.172251] LustreError: 141412:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2838.377163] Lustre: server umount lustre-MDT0001 complete [ 2843.842952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.111956] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2844.186792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2847.115284] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2849.251849] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.100340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2853.311579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2856.167797] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2861.218590] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2864.927328] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2869.254954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2869.473713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.901702] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2878.957808] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2882.542318] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2882.709336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2886.172671] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2886.338735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2893.517699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2893.574365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2893.578242] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2893.676988] Lustre: lustre-OST0001: new disk, initializing [ 2893.680556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2893.754645] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2897.171653] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 2902.856912] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2903.095674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2903.105298] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2904.146637] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2907.429820] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 2907.562853] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2911.289893] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95108612f800.ost_server_uuid 40 [ 2912.159557] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95108612f800.ost_server_uuid in FULL state after 0 sec [ 2918.886757] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2918.889872] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.896516] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2918.906099] Lustre: Skipped 1 previous similar message [ 2918.914053] Lustre: Skipped 1 previous similar message [ 2922.501169] Lustre: server umount lustre-OST0001 complete [ 2924.006894] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2924.022665] LustreError: Skipped 1 previous similar message [ 2926.048080] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2926.052288] LustreError: Skipped 1 previous similar message [ 2926.057386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.066482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2926.069828] Lustre: Skipped 1 previous similar message [ 2929.121726] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2929.123576] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.140065] LustreError: Skipped 1 previous similar message [ 2934.245350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2934.248427] Lustre: Skipped 2 previous similar messages [ 2934.249647] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2934.270821] LustreError: Skipped 1 previous similar message [ 2939.363521] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2939.505432] Lustre: server umount lustre-OST0000 complete [ 2944.482579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.487618] Lustre: Skipped 1 previous similar message [ 2948.400725] Lustre: server umount lustre-MDT0000 complete [ 2949.607247] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2949.625553] LustreError: Skipped 3 previous similar messages [ 2951.181253] LustreError: 147219:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313138 with bad export cookie 7531730738018072960 [ 2951.184962] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2951.194517] LustreError: 147219:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2965.548098] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2967.673454] Key type lgssc unregistered [ 2967.857987] LNet: 150341:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2967.867888] LNet: Removed LNI 192.168.203.153@tcp [ 2968.461690] Key type .llcrypt unregistered [ 2968.463263] Key type ._llcrypt unregistered [ 2993.083258] alg: No test for adler32 (adler32-zlib) [ 2993.837771] Key type ._llcrypt registered [ 2993.839423] Key type .llcrypt registered [ 2993.928262] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 3003.343222] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3004.189594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3004.462550] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 3004.661739] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 3004.664404] LNet: Accept secure, port 988 [ 3006.321197] Key type lgssc registered [ 3007.315716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3015.728618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3021.679916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3026.583858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3031.833725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3037.310250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3037.364303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.526928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3038.557423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3038.624313] Lustre: lustre-MDT0000: new disk, initializing [ 3038.698474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.717358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3041.764336] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 3050.291658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3050.361985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.450181] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3050.495944] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3050.499761] Lustre: Skipped 1 previous similar message [ 3050.565789] Lustre: lustre-MDT0001: new disk, initializing [ 3050.665990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3050.690518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3050.703495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3053.932256] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 3061.387392] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3065.587441] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3070.432075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3070.488856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.495255] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3070.699389] Lustre: lustre-OST0000: new disk, initializing [ 3070.704663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3070.815893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3074.031862] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 3078.169659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3078.173444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3080.564123] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3083.474352] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3083.610655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3086.349603] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3086.511649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3088.352744] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3088.353677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.358715] LustreError: Skipped 1 previous similar message [ 3088.368480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3088.377351] Lustre: Skipped 1 previous similar message [ 3088.382722] Lustre: Skipped 1 previous similar message [ 3093.475132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3093.478073] Lustre: Skipped 1 previous similar message [ 3098.595602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3098.601485] Lustre: Skipped 1 previous similar message [ 3102.175118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3102.317615] Lustre: server umount lustre-OST0000 complete [ 3107.299031] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3107.308046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.320078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3108.837540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3111.231698] Lustre: server umount lustre-MDT0000 complete [ 3113.636164] LustreError: 154429:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313300 with bad export cookie 14778635075840802892 [ 3113.643489] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3113.651941] LustreError: 154429:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3113.976800] Lustre: server umount lustre-MDT0001 complete [ 3119.626950] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 09:41:46 (1761313306) [ 3131.072940] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 3137.969391] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3146.372042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3152.113590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3156.911330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3161.719945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3167.740664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3167.802273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3168.042981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3168.103899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3168.241268] Lustre: lustre-MDT0000: new disk, initializing [ 3168.337441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.355486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3171.631209] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 3179.242040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3179.296854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3179.354855] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3179.397440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3179.402111] Lustre: Skipped 1 previous similar message [ 3179.466191] Lustre: lustre-MDT0001: new disk, initializing [ 3179.536502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3179.563334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3179.568159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3182.692167] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 3189.885638] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3193.546925] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3198.723378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3198.794793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3198.802586] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3198.978438] Lustre: lustre-OST0000: new disk, initializing [ 3198.980751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3199.044917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3201.812069] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 3207.719718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3207.736897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3209.280925] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3212.537581] Lustre: Failing over lustre-OST0000 [ 3212.739818] Lustre: server umount lustre-OST0000 complete [ 3220.447156] Lustre: 152661:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313400/real 1761313400] req@0000000023529e33 x1846870740598592/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761313407 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3220.447290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.478755] Lustre: 152661:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3220.485103] Lustre: Skipped 1 previous similar message [ 3222.495268] Lustre: 152659:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313402/real 1761313402] req@000000003af50e0c x1846870740598720/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1761313409 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3222.527512] Lustre: 152659:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3225.567144] Lustre: 152659:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313405/real 1761313405] req@00000000f2a7ef3d x1846870740599232/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761313412 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3225.592648] Lustre: 152659:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3230.333378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3230.604553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3230.621887] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3234.450585] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 3236.710252] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3238.270822] Lustre: Failing over lustre-MDT0000 [ 3238.296853] LustreError: 152657:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3238.461267] Lustre: server umount lustre-MDT0000 complete [ 3239.662214] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.724378] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.750714] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.153@tcp (at 0@lo) [ 3240.754517] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 3240.933934] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3240.950049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3241.952857] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3241.965181] LustreError: Skipped 3 previous similar messages [ 3245.776193] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3249.121903] Lustre: 152660:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313429/real 1761313429] req@000000003af50e0c x1846870740604864/t0(0) o400->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761313436 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3249.133188] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3251.926828] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.957923] LustreError: Skipped 7 previous similar messages [ 3255.817694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3260.385509] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.391703] LustreError: Skipped 10 previous similar messages [ 3266.535519] Lustre: Evicted from MGS (at 192.168.203.153@tcp) after server handle changed from 0xcd18424d089e83a2 to 0xcd18424d089e8872 [ 3266.561035] Lustre: MGC192.168.203.153@tcp: Connection restored to 192.168.203.153@tcp (at 0@lo) [ 3266.724513] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3266.783415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.819735] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3268.240604] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3270.252797] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 3272.172753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3272.190296] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3272.234089] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 3276.770075] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3276.778590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3276.787760] Lustre: Skipped 2 previous similar messages [ 3276.791372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3276.794752] Lustre: Skipped 2 previous similar messages [ 3278.696140] Lustre: server umount lustre-OST0000 complete [ 3282.403603] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3287.524252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3287.526855] Lustre: Skipped 5 previous similar messages [ 3290.771501] LustreError: 161417:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313478 with bad export cookie 14778635075840804978 [ 3290.774943] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3290.787783] LustreError: 161417:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3299.761915] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 3302.052691] Key type lgssc unregistered [ 3302.306598] LNet: 167274:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3302.319479] LNet: Removed LNI 192.168.203.153@tcp [ 3302.965812] Key type .llcrypt unregistered [ 3302.969347] Key type ._llcrypt unregistered [ 3320.179463] alg: No test for adler32 (adler32-zlib) [ 3320.935359] Key type ._llcrypt registered [ 3320.936947] Key type .llcrypt registered [ 3321.015801] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3321.547256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3321.948667] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 3322.098089] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 3322.100614] LNet: Accept secure, port 988 [ 3323.735800] Key type lgssc registered [ 3324.613508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3330.909506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.226345] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.270541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.882183] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 3337.704406] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.866444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.219697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3344.390202] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 3349.918409] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3354.008799] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3359.554781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3359.952678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.046763] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 3363.813898] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 3371.232248] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3388.667553] Lustre: lustre-MDT0000: Client d6131159-28ae-480d-aae7-f7ed372cc986 (at 192.168.203.53@tcp) reconnecting [ 3400.471553] Lustre: lustre-MDT0000: Client d6131159-28ae-480d-aae7-f7ed372cc986 (at 192.168.203.53@tcp) reconnecting [ 3446.013803] Lustre: lustre-MDT0000: Client d6131159-28ae-480d-aae7-f7ed372cc986 (at 192.168.203.53@tcp) reconnecting [ 3491.053945] Lustre: lustre-MDT0000: Client d6131159-28ae-480d-aae7-f7ed372cc986 (at 192.168.203.53@tcp) reconnecting [ 3545.314248] Lustre: lustre-MDT0000: Client d6131159-28ae-480d-aae7-f7ed372cc986 (at 192.168.203.53@tcp) reconnecting [ 3597.559611] Lustre: lustre-MDT0000: Client d6131159-28ae-480d-aae7-f7ed372cc986 (at 192.168.203.53@tcp) reconnecting [ 3646.692234] Lustre: lustre-MDT0000: Client d6131159-28ae-480d-aae7-f7ed372cc986 (at 192.168.203.53@tcp) reconnecting [ 3699.934546] Lustre: lustre-MDT0000: Client d6131159-28ae-480d-aae7-f7ed372cc986 (at 192.168.203.53@tcp) reconnecting [ 3822.872738] Lustre: lustre-MDT0000: Client d6131159-28ae-480d-aae7-f7ed372cc986 (at 192.168.203.53@tcp) reconnecting [ 3822.884663] Lustre: Skipped 1 previous similar message [ 3970.297793] Lustre: lustre-MDT0000: Client d6131159-28ae-480d-aae7-f7ed372cc986 (at 192.168.203.53@tcp) reconnecting [ 3970.306579] Lustre: Skipped 2 previous similar messages [ 4265.218870] Lustre: lustre-MDT0000: Client d6131159-28ae-480d-aae7-f7ed372cc986 (at 192.168.203.53@tcp) reconnecting [ 4265.227831] Lustre: Skipped 5 previous similar messages [ 4644.726536] Lustre: Failing over lustre-MDT0000 [ 4644.963726] Lustre: server umount lustre-MDT0000 complete [ 4646.370292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4646.376796] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4646.428752] Lustre: Skipped 2 previous similar messages [ 4646.485386] LustreError: Skipped 1 previous similar message [ 4651.492908] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.521232] LustreError: Skipped 3 previous similar messages [ 4656.611174] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.623912] LustreError: Skipped 3 previous similar messages [ 4661.728944] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4661.746158] LustreError: Skipped 3 previous similar messages [ 4661.781937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4661.863307] LustreError: 11-0: MGC192.168.203.153@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 4661.865750] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4661.887792] Lustre: Evicted from MGS (at 192.168.203.153@tcp) after server handle changed from 0x7274b25db22fa42d to 0x7274b25db2321047 [ 4661.893753] Lustre: MGC192.168.203.153@tcp: Connection restored to 192.168.203.153@tcp (at 0@lo) [ 4661.967121] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.53@tcp (not set up) [ 4662.052765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.106769] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4665.218552] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 4667.088613] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4667.377273] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4667.398480] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4667.417932] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 4671.940744] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4673.083105] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4692.961526] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4692.966480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.968898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4692.974100] Lustre: Skipped 2 previous similar messages [ 4692.985624] Lustre: Skipped 1 previous similar message [ 4694.689860] Lustre: server umount lustre-MDT0000 complete [ 4697.486489] LustreError: 168717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314884 with bad export cookie 8247412933113548871 [ 4697.490912] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4697.513203] LustreError: 168717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4698.080480] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4698.080542] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4698.093683] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4698.096466] Lustre: Skipped 1 previous similar message [ 4702.117294] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4703.981553] Lustre: server umount lustre-MDT0001 complete [ 4712.798592] Lustre: server umount lustre-OST0000 complete [ 4719.758669] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 4725.605624] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4732.083696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4736.709337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4741.346576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4745.985726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4750.875170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4750.921688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.093436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4751.126264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4751.185908] Lustre: lustre-MDT0000: new disk, initializing [ 4751.277708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4751.294305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4754.294889] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 4761.547052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4761.595932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.647031] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4761.673872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4761.676584] Lustre: Skipped 1 previous similar message [ 4761.731916] Lustre: lustre-MDT0001: new disk, initializing [ 4761.788138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4761.809324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4761.816553] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4764.413767] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 4770.471777] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4773.562344] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4777.683936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4777.749333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4777.755791] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4777.923881] Lustre: lustre-OST0000: new disk, initializing [ 4777.926793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4778.005379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4781.294719] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 4787.284922] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4787.746469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4787.751207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4789.943775] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4790.123064] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4792.657414] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4792.784218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4797.922606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4797.922733] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4797.929856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4797.936508] Lustre: Skipped 2 previous similar messages [ 4808.159124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4808.165581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4808.165589] Lustre: Skipped 2 previous similar messages [ 4808.316672] Lustre: server umount lustre-OST0000 complete [ 4813.280030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.284895] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4813.294619] Lustre: Skipped 1 previous similar message [ 4825.056040] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4825.170660] Lustre: server umount lustre-MDT0000 complete [ 4827.568304] LustreError: 175786:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315014 with bad export cookie 8247412933113549592 [ 4827.573821] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4827.597035] LustreError: 175786:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4833.986450] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:10:21 (1761315021) [ 4846.215397] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 4852.383616] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4860.153837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4865.054733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4869.230925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4873.691299] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4879.627258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4879.686701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.871464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4879.894920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4879.972955] Lustre: lustre-MDT0000: new disk, initializing [ 4880.052221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4880.071204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4883.647553] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 4892.149484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4892.207934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4892.286459] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4892.322968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4892.325879] Lustre: Skipped 1 previous similar message [ 4892.399703] Lustre: lustre-MDT0001: new disk, initializing [ 4892.479418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4892.535289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4892.550850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4895.727243] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 4902.878405] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4907.134789] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4912.431739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4912.479818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4912.482485] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4912.700302] Lustre: lustre-OST0000: new disk, initializing [ 4912.707597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4912.797195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4915.776506] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 4921.908377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4921.916583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4922.861576] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4928.996703] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4929.000499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4929.007467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4929.009961] Lustre: Skipped 7 previous similar messages [ 4941.279117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4941.430339] Lustre: server umount lustre-OST0000 complete [ 4941.432639] Lustre: Skipped 1 previous similar message [ 4943.840872] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4943.844235] LustreError: Skipped 1 previous similar message [ 4947.426705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4947.435954] Lustre: Skipped 2 previous similar messages [ 4958.175241] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4960.504520] LustreError: 182772:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315147 with bad export cookie 8247412933113550446 [ 4960.505192] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4960.511344] LustreError: 182772:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4965.777058] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:12:33 (1761315153) [ 4975.258964] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 4979.535116] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4985.348963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4989.112226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4993.028127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4997.608651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5001.933454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5001.973850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.121544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5002.148339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5002.197981] Lustre: lustre-MDT0000: new disk, initializing [ 5002.241692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.253506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5004.969874] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5011.951786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5012.002118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5012.078325] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5012.118688] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5012.133255] LustreError: Skipped 2 previous similar messages [ 5012.289839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5015.400969] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5023.442696] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5027.770405] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5033.124513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5033.197118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5033.205407] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5036.466447] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5041.164463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5041.168603] Lustre: Skipped 1 previous similar message [ 5041.171390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5041.615457] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5046.240485] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5046.244309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.246507] LustreError: Skipped 1 previous similar message [ 5046.260580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5046.266531] Lustre: Skipped 12 previous similar messages [ 5047.281799] Lustre: server umount lustre-OST0000 complete [ 5047.284100] Lustre: Skipped 2 previous similar messages [ 5056.481277] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5056.489514] LustreError: Skipped 1 previous similar message [ 5058.616435] LustreError: 189235:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315245 with bad export cookie 8247412933113551524 [ 5058.619544] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5058.626026] LustreError: 189235:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5065.734693] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 5067.381915] Key type lgssc unregistered [ 5067.564602] LNet: 193199:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5067.572654] LNet: Removed LNI 192.168.203.153@tcp [ 5067.968155] Key type .llcrypt unregistered [ 5067.970844] Key type ._llcrypt unregistered [ 5083.191090] alg: No test for adler32 (adler32-zlib) [ 5083.940711] Key type ._llcrypt registered [ 5083.942287] Key type .llcrypt registered [ 5084.022511] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 5084.721630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5084.939488] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 5085.046528] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 5085.049013] LNet: Accept secure, port 988 [ 5086.663561] Key type lgssc registered [ 5087.413760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5092.726465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5094.047740] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5094.114405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5097.252441] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5099.490169] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5103.161357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5103.443321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5106.563316] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5112.138033] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5116.419135] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5121.582588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5121.976104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5124.035148] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 5125.641804] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5131.053265] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5153.762362] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5153.764528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5153.771786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5153.777700] Lustre: Skipped 1 previous similar message [ 5153.788167] Lustre: Skipped 1 previous similar message [ 5155.914533] Lustre: server umount lustre-OST0000 complete [ 5158.880995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5158.886405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5158.894727] Lustre: Skipped 1 previous similar message [ 5164.004266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5164.007080] Lustre: Skipped 2 previous similar messages [ 5169.128781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5173.217173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5173.298721] Lustre: server umount lustre-MDT0000 complete [ 5175.270241] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.279357] LustreError: Skipped 1 previous similar message [ 5175.862286] LustreError: 194640:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315363 with bad export cookie 1609663991569274357 [ 5175.863561] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5175.867219] LustreError: 194640:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5176.080317] Lustre: server umount lustre-MDT0001 complete [ 5183.779603] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 5185.868763] Key type lgssc unregistered [ 5186.052847] LNet: 198336:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5186.059136] LNet: Removed LNI 192.168.203.153@tcp [ 5186.589862] Key type .llcrypt unregistered [ 5186.592370] Key type ._llcrypt unregistered [ 5201.682178] alg: No test for adler32 (adler32-zlib) [ 5202.433625] Key type ._llcrypt registered [ 5202.437471] Key type .llcrypt registered [ 5202.520936] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 5203.533700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5203.851276] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 5204.050028] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 5204.065556] LNet: Accept secure, port 988 [ 5205.727080] Key type lgssc registered [ 5206.976285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5213.945247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5215.239936] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5215.299840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.754325] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5220.324343] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5225.445904] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5225.565069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.892864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5229.843359] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5236.541522] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5242.193775] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5247.187808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5247.427323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5250.540750] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5255.486107] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5264.817289] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5269.986821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5269.993189] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5269.995797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5271.260887] Lustre: lustre-OST0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 5271.264239] Lustre: Skipped 1 previous similar message [ 5275.109546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5275.114814] Lustre: Skipped 1 previous similar message [ 5275.568269] Lustre: server umount lustre-OST0000 complete [ 5280.706344] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9510869ff000.ost_server_uuid 40 [ 5282.018279] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5295.587484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.592756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.593887] Lustre: Skipped 2 previous similar messages [ 5300.865891] Lustre: server umount lustre-MDT0000 complete [ 5301.729078] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5301.747739] LustreError: Skipped 1 previous similar message [ 5304.434410] LustreError: 199780:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315491 with bad export cookie 12753101535171178446 [ 5304.438446] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5304.447767] LustreError: 199780:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5304.829558] Lustre: server umount lustre-MDT0001 complete [ 5311.262450] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:18:18 (1761315498) [ 5314.998305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.331558] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5315.395672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5318.116857] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5323.237771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.489201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5325.778148] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5330.300214] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5334.325522] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5339.159725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5339.401093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5342.764385] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5346.607701] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 5348.879503] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5353.684854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5353.747334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5353.761284] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5353.859066] Lustre: lustre-OST0001: new disk, initializing [ 5353.865266] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5356.522183] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5362.833406] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5363.802210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5363.816944] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5367.265580] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5367.272447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.287361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5367.290882] Lustre: Skipped 3 previous similar messages [ 5368.289848] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5370.756662] Lustre: server umount lustre-OST0000 complete [ 5373.650698] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.671889] LustreError: Skipped 1 previous similar message [ 5376.134350] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5376.309507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5379.926547] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5380.109282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5389.794760] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5389.810180] LustreError: Skipped 10 previous similar messages [ 5394.913793] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5394.916750] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.919948] LustreError: Skipped 1 previous similar message [ 5394.920612] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5394.930809] Lustre: Skipped 2 previous similar messages [ 5394.933026] Lustre: Skipped 4 previous similar messages [ 5398.437437] Lustre: server umount lustre-OST0001 complete [ 5405.153483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5405.159734] Lustre: Skipped 1 previous similar message [ 5407.375465] Lustre: server umount lustre-MDT0000 complete [ 5410.180116] LustreError: 206717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315597 with bad export cookie 12753101535171179517 [ 5410.182258] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5410.186516] LustreError: 206717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5414.473516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5417.239970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5419.986661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5422.880966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5427.484684] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:20:14 (1761315614) [ 5433.742420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5434.097834] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5434.135224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5434.405697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5434.410654] Lustre: Skipped 1 previous similar message [ 5438.159130] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5444.533814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5444.615911] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5444.665485] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5448.011718] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5453.489693] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5458.074757] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5462.648066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5462.766370] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5466.264764] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5471.650794] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5472.349795] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 5476.863125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5476.926674] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5477.091531] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5477.094375] Lustre: Skipped 2 previous similar messages [ 5480.367337] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5484.898816] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5487.583968] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5487.587761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5487.600849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5487.603500] Lustre: Skipped 3 previous similar messages [ 5492.859320] Lustre: server umount lustre-OST0000 complete [ 5492.873611] Lustre: Skipped 1 previous similar message [ 5495.505576] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5495.517207] LustreError: Skipped 2 previous similar messages [ 5511.139710] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5511.148583] LustreError: Skipped 1 previous similar message [ 5511.151363] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5511.159851] Lustre: Skipped 1 previous similar message [ 5521.888059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5521.892526] Lustre: Skipped 9 previous similar messages [ 5525.142605] LustreError: 213190:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315712 with bad export cookie 12753101535171180896 [ 5525.143186] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5525.149426] LustreError: 213190:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5525.399158] Lustre: server umount lustre-MDT0001 complete [ 5525.401136] Lustre: Skipped 2 previous similar messages [ 5529.498170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5532.419102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5535.426029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5538.207848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5542.980991] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:22:10 (1761315730) [ 5556.277248] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 5562.024280] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 5568.367554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5573.163917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5577.965364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5583.165360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5588.742360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5588.894070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5589.179486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5589.202898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5589.312061] Lustre: lustre-MDT0000: new disk, initializing [ 5589.386914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5589.404271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5593.461427] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5600.729989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5600.812336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5600.887052] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5600.916374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5600.919898] Lustre: Skipped 1 previous similar message [ 5600.982170] Lustre: lustre-MDT0001: new disk, initializing [ 5601.114364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5601.135398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5604.855801] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5613.145253] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5618.475712] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5623.262233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5623.329938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5623.334967] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5623.559410] Lustre: lustre-OST0000: new disk, initializing [ 5623.561829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5627.527392] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5631.538959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5631.542635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5635.640797] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5639.521166] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5639.754863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5643.385775] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5643.584376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5646.816101] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5646.823972] LustreError: Skipped 2 previous similar messages [ 5646.828856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.839269] Lustre: Skipped 3 previous similar messages [ 5646.844965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5651.227015] Lustre: server umount lustre-OST0000 complete [ 5662.690538] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5662.706620] LustreError: Skipped 10 previous similar messages [ 5663.495155] LustreError: 219309:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315850 with bad export cookie 12753101535171182415 [ 5663.499150] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5663.504420] LustreError: 219309:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5670.672742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.710882] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5681.707190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.478651] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5692.065664] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5697.154217] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5701.734943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5704.692685] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5710.657183] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5714.376856] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5721.949508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 5725.400864] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5725.580321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5731.308894] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5734.879165] Lustre: 199110:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315915/real 1761315915] req@00000000c1ba3405 x1846873047534976/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761315922 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 5734.910680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.919892] Lustre: Skipped 3 previous similar messages [ 5738.975349] Lustre: 199111:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315919/real 1761315919] req@00000000212548a6 x1846873047535552/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761315926 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 5739.999323] Lustre: 199110:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315919/real 1761315919] req@00000000931971ad x1846873047535232/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761315926 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5742.049400] Lustre: 199113:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315922/real 1761315922] req@000000007d4f85a0 x1846873047535872/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761315929 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5742.093778] Lustre: 199113:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5743.079190] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5770.256051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5770.534898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.546350] Lustre: Skipped 5 previous similar messages [ 5772.593116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.609112] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 5772.619385] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 5774.014281] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5779.592517] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5780.879364] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5784.036701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5784.045460] Lustre: Skipped 4 previous similar messages [ 5789.751126] Lustre: server umount lustre-OST0000 complete [ 5789.753086] Lustre: Skipped 3 previous similar messages [ 5799.392854] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5799.399704] LustreError: Skipped 4 previous similar messages [ 5801.890121] LustreError: 222797:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315989 with bad export cookie 12753101535171183262 [ 5801.896051] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5801.905532] LustreError: 222797:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5809.483853] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:26:36 (1761315996) [ 5814.882198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5818.440328] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5825.335473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5828.611992] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5834.398181] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5839.227859] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5844.341860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5848.001474] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5854.085389] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5857.546849] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5857.776423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5861.348505] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5861.572784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5867.240744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5867.295971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5867.300902] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5867.391471] Lustre: lustre-OST0001: new disk, initializing [ 5867.393740] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5870.273604] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5876.303194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5876.309984] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5877.334231] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5881.081598] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 5881.325845] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5885.238663] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 5885.430577] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5892.071722] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5892.075264] LustreError: Skipped 2 previous similar messages [ 5892.075561] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5892.103928] Lustre: Skipped 5 previous similar messages [ 5901.281276] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5906.438642] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 5906.605989] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5910.096660] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 5910.282201] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5937.676471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5940.947838] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 5943.271131] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5943.286386] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5943.286575] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to (at 0@lo) [ 5943.311556] Lustre: Skipped 1 previous similar message [ 5947.087430] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5948.249260] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5972.965046] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5972.968129] LustreError: Skipped 2 previous similar messages [ 5986.909755] LustreError: 229089:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316174 with bad export cookie 12753101535171184452 [ 5986.912994] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5986.914135] LustreError: 229089:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5997.436316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6000.162856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6002.785541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6005.738588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6010.205885] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:29:57 (1761316197) [ 6014.187108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.365984] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6014.382109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6016.808226] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6021.892451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6021.969792] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6024.930784] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6029.149576] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6033.652336] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6038.365043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6038.518349] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6038.642602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.645603] Lustre: Skipped 7 previous similar messages [ 6041.312207] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6046.306189] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6057.942637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6057.999663] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6061.253239] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6067.383632] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6068.639332] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6071.640214] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6071.769283] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6074.859668] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6075.034825] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6079.564162] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9510868f1000.ost_server_uuid 40 [ 6080.756974] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9510868f1000.ost_server_uuid in FULL state after 0 sec [ 6081.590476] Lustre: Permanently deactivating lustre-OST0001 [ 6081.597087] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6081.600917] Lustre: Skipped 1 previous similar message [ 6083.734897] Lustre: Permanently reactivating lustre-OST0001 [ 6087.147408] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6087.151458] Lustre: Skipped 18 previous similar messages [ 6092.031048] Lustre: server umount lustre-OST0001 complete [ 6092.039850] Lustre: Skipped 7 previous similar messages [ 6094.818213] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.830191] LustreError: Skipped 33 previous similar messages [ 6104.034721] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6104.039135] LustreError: Skipped 3 previous similar messages [ 6111.235459] LustreError: 238015:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316298 with bad export cookie 12753101535171185726 [ 6111.236193] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6111.241289] LustreError: 238015:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6114.802952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6117.186322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6119.781041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6122.920571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6128.468093] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:31:55 (1761316315) [ 6129.588432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6135.636531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6136.096291] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6136.133790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6136.139567] Lustre: Skipped 5 previous similar messages [ 6139.828050] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6145.486437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.541766] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6148.974435] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6154.416821] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6157.883574] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6162.398201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6162.448197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6162.603071] Lustre: Permanently deactivating lustre-OST0000 [ 6165.457690] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6170.784286] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6172.214850] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6172.288906] LustreError: 247530:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6176.371727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6179.850919] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6180.115910] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 6186.127065] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6187.494460] Lustre: Permanently reactivating lustre-OST0000 [ 6187.504736] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 6187.508903] Lustre: Skipped 4 previous similar messages [ 6187.720689] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6187.764817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6187.773817] Lustre: Skipped 12 previous similar messages [ 6187.778981] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6187.786164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.804950] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 6187.811962] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 6227.247086] LustreError: 244859:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316414 with bad export cookie 12753101535171187315 [ 6227.250649] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6227.255358] LustreError: 244859:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6235.467515] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6237.240182] Key type lgssc unregistered [ 6237.437525] LNet: 249819:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6237.447802] LNet: Removed LNI 192.168.203.153@tcp [ 6237.997681] Key type .llcrypt unregistered [ 6237.999249] Key type ._llcrypt unregistered [ 6252.919604] alg: No test for adler32 (adler32-zlib) [ 6253.675162] Key type ._llcrypt registered [ 6253.676778] Key type .llcrypt registered [ 6253.785363] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6254.600781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6254.927238] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 6255.068758] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6255.071782] LNet: Accept secure, port 988 [ 6256.711578] Key type lgssc registered [ 6257.446649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6260.379429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6265.111910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.404892] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6266.449562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6269.460297] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6271.461550] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6275.219850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6275.274660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.351153] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6275.379588] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6275.384402] Lustre: Permanently deactivating lustre-MDT0001 [ 6275.574615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6278.504354] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6280.197086] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6280.208822] LustreError: 251364:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6280.222200] LustreError: 251364:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6280.227109] LustreError: 251364:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 6283.111190] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6286.554593] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6291.970653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6292.192436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6295.534798] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6296.299550] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6296.312837] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 6301.664809] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6302.493341] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6305.766870] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.778561] LustreError: Skipped 1 previous similar message [ 6305.783517] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:131 to 0x280000400:161 [ 6308.612099] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6308.814538] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6312.717025] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6313.968634] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 [ 6313.972637] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:129 [ 6318.564808] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6321.099957] Lustre: Permanently reactivating lustre-MDT0001 [ 6321.110584] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 6321.113705] Lustre: Skipped 3 previous similar messages [ 6325.491485] Lustre: lustre-MDT0001: Client fb42d1bd-5bad-417b-a7a7-f232cedbd266 (at 192.168.203.53@tcp) reconnecting [ 6331.360885] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6332.585238] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6336.631526] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 6336.777632] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6339.905587] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 6340.061883] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6341.594526] Lustre: Permanently deactivating lustre-MDT0001 [ 6341.605472] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 6341.608404] Lustre: Skipped 2 previous similar messages [ 6350.961977] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6358.921762] LustreError: 256119:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (0000000040de3e56) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 6360.037819] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.040419] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6360.043545] Lustre: Skipped 1 previous similar message [ 6360.052904] LustreError: Skipped 3 previous similar messages [ 6361.022303] Lustre: server umount lustre-MDT0000 complete [ 6364.638209] LustreError: 254907:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316551 with bad export cookie 194956669127265006 [ 6364.640365] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6364.654463] LustreError: 254907:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6365.030079] Lustre: server umount lustre-MDT0001 complete [ 6368.178229] Lustre: server umount lustre-OST0000 complete [ 6376.502161] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:36:03 (1761316563) [ 6384.157468] Lustre: server umount lustre-OST0001 complete [ 6388.911320] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 6394.187440] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6399.678793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6403.771749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6407.518243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6412.259832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6417.317724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6417.364094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6417.499898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6417.503732] Lustre: Skipped 2 previous similar messages [ 6417.529711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6417.585394] Lustre: lustre-MDT0000: new disk, initializing [ 6417.700913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6417.715268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6420.455743] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6426.896554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6426.939451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6427.013431] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6427.050814] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6427.056960] Lustre: Skipped 1 previous similar message [ 6427.059626] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6427.107762] Lustre: lustre-MDT0001: new disk, initializing [ 6427.162124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6427.197393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6427.201563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6430.018323] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6435.983333] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6439.439956] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6443.376310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6443.409898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6443.413283] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6443.543630] Lustre: lustre-OST0000: new disk, initializing [ 6443.546931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6443.597459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6445.794624] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6451.539345] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6453.784513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6453.791153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6458.018984] LustreError: 261406:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6464.258899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6464.305752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6464.308922] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6464.438860] Lustre: lustre-OST0001: new disk, initializing [ 6464.452748] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6467.373275] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6468.111109] LustreError: 261406:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 6468.130052] LustreError: 261406:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6470.696571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6470.700786] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6474.265733] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6475.345691] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6478.183086] LustreError: 261406:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 6480.866343] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6480.868513] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6480.871742] LustreError: Skipped 1 previous similar message [ 6480.878160] Lustre: Skipped 2 previous similar messages [ 6480.878964] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6480.897134] Lustre: Skipped 1 previous similar message [ 6483.172041] Lustre: lustre-OST0001: Not available for connect from 192.168.203.53@tcp (stopping) [ 6485.572553] Lustre: server umount lustre-OST0001 complete [ 6485.991472] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6485.998862] LustreError: Skipped 1 previous similar message [ 6499.808344] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6499.811596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6499.814843] LustreError: Skipped 1 previous similar message [ 6499.820652] Lustre: Skipped 1 previous similar message [ 6499.821314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6499.832307] Lustre: Skipped 1 previous similar message [ 6505.463988] Lustre: server umount lustre-OST0000 complete [ 6509.024438] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6509.027304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.038814] Lustre: Skipped 1 previous similar message [ 6509.042450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6509.047976] Lustre: Skipped 2 previous similar messages [ 6516.939418] LustreError: 259818:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316704 with bad export cookie 194956669127267785 [ 6516.941867] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6516.946901] LustreError: 259818:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6523.207737] Lustre: server umount lustre-MDT0001 complete [ 6523.209827] Lustre: Skipped 1 previous similar message [ 6531.203635] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6533.336471] Key type lgssc unregistered [ 6533.551579] LNet: 265022:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6533.557783] LNet: Removed LNI 192.168.203.153@tcp [ 6534.070820] Key type .llcrypt unregistered [ 6534.074987] Key type ._llcrypt unregistered [ 6558.484397] alg: No test for adler32 (adler32-zlib) [ 6559.239589] Key type ._llcrypt registered [ 6559.245428] Key type .llcrypt registered [ 6559.404550] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 6570.358479] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6571.083116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6571.327176] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 6571.513037] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6571.515694] LNet: Accept secure, port 988 [ 6573.159635] Key type lgssc registered [ 6574.012545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6581.145087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6586.493300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6590.963035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6595.715834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6601.343423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6601.422256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6602.643056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6602.684493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6602.779653] Lustre: lustre-MDT0000: new disk, initializing [ 6602.884271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6602.920429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6607.004813] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6615.139246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6615.190074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6615.273224] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6615.306709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6615.309830] Lustre: Skipped 1 previous similar message [ 6615.369709] Lustre: lustre-MDT0001: new disk, initializing [ 6615.475772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6615.510687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6615.518848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6618.850838] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6626.143580] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6629.956387] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6634.397212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6634.447524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6634.451535] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6634.623325] Lustre: lustre-OST0000: new disk, initializing [ 6634.626685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6634.695446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6637.807476] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6640.182961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6640.198490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6644.853392] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6647.995840] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6648.161814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6651.324324] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6651.512833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6655.970952] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6655.972167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6655.982230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6655.987783] Lustre: Skipped 1 previous similar message [ 6655.999955] Lustre: Skipped 1 previous similar message [ 6659.408394] Lustre: server umount lustre-OST0000 complete [ 6666.211036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6666.218521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6666.230236] Lustre: Skipped 1 previous similar message [ 6668.744166] Lustre: server umount lustre-MDT0000 complete [ 6671.332850] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6671.349754] LustreError: Skipped 1 previous similar message [ 6671.732848] LustreError: 269113:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316859 with bad export cookie 13596526893818174220 [ 6671.736262] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6671.737895] LustreError: 269113:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6672.139490] Lustre: server umount lustre-MDT0001 complete [ 6678.987334] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 10:41:06 (1761316866) [ 6683.726686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6684.039316] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6684.099403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6686.901457] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6689.248952] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6691.551698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.785514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6694.579793] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6699.586996] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6703.725992] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6708.266659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6708.489066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6711.293192] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6717.235033] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6735.332753] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6735.339804] LustreError: Skipped 1 previous similar message [ 6735.348651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6735.373353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6735.376208] Lustre: Skipped 1 previous similar message [ 6737.377312] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6737.383795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6737.390058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6738.978726] Lustre: server umount lustre-OST0000 complete [ 6742.608056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6751.832685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6752.125335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6753.129996] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6753.142046] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 6753.146838] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 6755.183405] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6758.245219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6758.264505] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 6758.268111] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 6760.569415] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6768.610079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6768.614576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6768.615083] Lustre: Skipped 1 previous similar message [ 6768.615393] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6768.618226] Lustre: Skipped 2 previous similar messages [ 6773.446056] Lustre: server umount lustre-OST0000 complete [ 6778.849289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6778.859501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6778.863210] Lustre: Skipped 1 previous similar message [ 6782.301398] Lustre: server umount lustre-MDT0000 complete [ 6783.969048] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6783.987515] LustreError: Skipped 1 previous similar message [ 6785.543230] LustreError: 276386:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316972 with bad export cookie 13596526893818175074 [ 6785.546222] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6785.553982] LustreError: 276386:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6795.948847] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6798.249713] Key type lgssc unregistered [ 6798.567488] LNet: 279097:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6798.577293] LNet: Removed LNI 192.168.203.153@tcp [ 6799.324441] Key type .llcrypt unregistered [ 6799.326508] Key type ._llcrypt unregistered [ 6822.567951] alg: No test for adler32 (adler32-zlib) [ 6823.320372] Key type ._llcrypt registered [ 6823.322505] Key type .llcrypt registered [ 6823.388095] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6825.124319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6825.581627] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 6825.798655] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6825.801700] LNet: Accept secure, port 988 [ 6827.528747] Key type lgssc registered [ 6828.515286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6836.163763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6837.540756] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6837.570894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6841.306980] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6842.849108] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.979382] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6848.257419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.506982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6851.776557] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6857.694874] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6862.325241] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6868.365437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6868.673291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6872.426961] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6878.830652] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6888.103255] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 6908.896814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6908.901149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6908.920875] Lustre: Skipped 1 previous similar message [ 6913.026557] Lustre: server umount lustre-OST0000 complete [ 6919.136863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6919.142581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6919.147023] Lustre: Skipped 1 previous similar message [ 6919.149425] Lustre: Skipped 1 previous similar message [ 6921.865202] Lustre: server umount lustre-MDT0000 complete [ 6924.258107] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6924.266362] LustreError: Skipped 1 previous similar message [ 6924.437661] LustreError: 280561:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317111 with bad export cookie 6872475677932155957 [ 6924.440736] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6924.444569] LustreError: 280561:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6924.643030] Lustre: server umount lustre-MDT0001 complete [ 6933.539915] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6935.560599] Key type lgssc unregistered [ 6935.779471] LNet: 285031:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6935.791661] LNet: Removed LNI 192.168.203.153@tcp [ 6936.458448] Key type .llcrypt unregistered [ 6936.466237] Key type ._llcrypt unregistered [ 6948.029162] alg: No test for adler32 (adler32-zlib) [ 6948.785560] Key type ._llcrypt registered [ 6948.790238] Key type .llcrypt registered [ 6948.920935] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6950.053969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6950.362877] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 6950.574361] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6950.577163] LNet: Accept secure, port 988 [ 6952.223873] Key type lgssc registered [ 6953.017810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6958.423694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6959.694020] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6959.732286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6962.651262] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6965.218527] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6969.637414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.034185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6973.435472] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6979.148121] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6983.588532] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6988.390418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6988.641626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6989.674814] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 6991.770446] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 6996.964566] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7008.743873] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7008.761222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7008.778426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7010.273562] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7010.275230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7010.279224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7010.294346] Lustre: Skipped 1 previous similar message [ 7014.958893] Lustre: server umount lustre-OST0000 complete [ 7020.514393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7020.516493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7020.530894] Lustre: Skipped 1 previous similar message [ 7020.537445] Lustre: Skipped 1 previous similar message [ 7025.632572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7025.640746] Lustre: Skipped 1 previous similar message [ 7031.780975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7031.788954] Lustre: Skipped 3 previous similar messages [ 7032.287119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7032.398068] Lustre: server umount lustre-MDT0000 complete [ 7035.962719] LustreError: 286279:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317223 with bad export cookie 18357842976713107004 [ 7035.967971] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7035.982614] LustreError: 286279:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7036.322650] Lustre: server umount lustre-MDT0001 complete [ 7045.304130] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 7047.379786] Key type lgssc unregistered [ 7047.546344] LNet: 290126:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7047.553379] LNet: Removed LNI 192.168.203.153@tcp [ 7048.103991] Key type .llcrypt unregistered [ 7048.107743] Key type ._llcrypt unregistered [ 7058.616441] alg: No test for adler32 (adler32-zlib) [ 7059.371415] Key type ._llcrypt registered [ 7059.373375] Key type .llcrypt registered [ 7059.434111] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7060.019889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7060.225112] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 7060.348723] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 7060.352058] LNet: Accept secure, port 988 [ 7061.959266] Key type lgssc registered [ 7062.500048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7066.469540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7067.726382] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.776063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7070.273641] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7073.249397] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7075.650366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7075.909357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7078.363480] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7083.847140] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7087.892412] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7093.421473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7093.904925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7096.965635] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7098.032795] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 7102.995284] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7112.674413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7112.675199] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7112.675217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7112.680265] Lustre: Skipped 1 previous similar message [ 7117.794337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7117.796698] Lustre: Skipped 1 previous similar message [ 7118.468388] Lustre: server umount lustre-OST0000 complete [ 7121.893750] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7121.897884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7121.904968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7122.916463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7127.313202] Lustre: server umount lustre-MDT0000 complete [ 7128.033440] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7128.043805] LustreError: Skipped 1 previous similar message [ 7130.213608] LustreError: 291372:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317317 with bad export cookie 8137305249138452606 [ 7130.222252] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7130.493399] Lustre: server umount lustre-MDT0001 complete [ 7140.450421] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 7142.447359] Key type lgssc unregistered [ 7142.669723] LNet: 295045:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7142.675546] LNet: Removed LNI 192.168.203.153@tcp [ 7143.368931] Key type .llcrypt unregistered [ 7143.376562] Key type ._llcrypt unregistered [ 7168.593519] alg: No test for adler32 (adler32-zlib) [ 7169.367047] Key type ._llcrypt registered [ 7169.368899] Key type .llcrypt registered [ 7169.525537] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7170.714975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7171.170132] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 7171.322171] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 7171.324870] LNet: Accept secure, port 988 [ 7172.968744] Key type lgssc registered [ 7173.886757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7180.126897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7181.536162] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7181.588941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7185.570161] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7186.911714] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7192.037951] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7192.480507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7192.803464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7196.835638] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7203.003543] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7207.282528] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7212.817401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7213.090631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7216.557903] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7222.918360] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7232.562259] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 7252.959929] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7252.965559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.974956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7252.977942] Lustre: Skipped 1 previous similar message [ 7255.552697] Lustre: server umount lustre-OST0000 complete [ 7259.616237] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7259.632814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7259.651445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7259.658369] Lustre: Skipped 1 previous similar message [ 7263.206526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7263.215134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7263.224197] Lustre: Skipped 1 previous similar message [ 7265.016587] Lustre: server umount lustre-MDT0000 complete [ 7268.137241] LustreError: 296512:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317455 with bad export cookie 16430394112241042118 [ 7268.139104] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7268.144901] LustreError: 296512:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7268.367034] Lustre: server umount lustre-MDT0001 complete [ 7279.271694] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 7281.619798] Key type lgssc unregistered [ 7281.794749] LNet: 301028:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7281.801103] LNet: Removed LNI 192.168.203.153@tcp [ 7282.429555] Key type .llcrypt unregistered [ 7282.431282] Key type ._llcrypt unregistered [ 7294.776715] alg: No test for adler32 (adler32-zlib) [ 7295.531417] Key type ._llcrypt registered [ 7295.533368] Key type .llcrypt registered [ 7295.613686] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7296.351756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7296.597216] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 7296.744676] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 7296.750815] LNet: Accept secure, port 988 [ 7298.399138] Key type lgssc registered [ 7299.192492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7304.845493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7306.238242] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7306.291267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7309.882533] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7311.333946] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7316.449035] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7316.640803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.916895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7320.362890] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7325.461779] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7329.437108] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7334.049978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7334.288894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7336.365670] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 7337.346545] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7343.055408] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7355.880143] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7355.883343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7355.912425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7357.923280] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7357.924205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7357.936585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.952072] Lustre: Skipped 1 previous similar message [ 7361.117645] Lustre: server umount lustre-OST0000 complete [ 7368.164144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7368.186795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7373.282045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7373.290314] Lustre: Skipped 2 previous similar messages [ 7378.399115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7378.475280] Lustre: server umount lustre-MDT0000 complete [ 7379.425394] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7379.432478] LustreError: Skipped 1 previous similar message [ 7380.903410] LustreError: 302278:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317568 with bad export cookie 11481768436519892361 [ 7380.906360] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7380.912751] LustreError: 302278:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7381.150775] Lustre: server umount lustre-MDT0001 complete [ 7388.858229] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 7390.497680] Key type lgssc unregistered [ 7390.655642] LNet: 306133:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7390.660540] LNet: Removed LNI 192.168.203.153@tcp [ 7391.142164] Key type .llcrypt unregistered [ 7391.144119] Key type ._llcrypt unregistered [ 7402.699499] alg: No test for adler32 (adler32-zlib) [ 7403.451472] Key type ._llcrypt registered [ 7403.453598] Key type .llcrypt registered [ 7403.535905] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7404.270348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7404.521237] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 7404.628245] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 7404.630727] LNet: Accept secure, port 988 [ 7406.255161] Key type lgssc registered [ 7407.137994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7412.743297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7414.129912] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7414.191758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.217521] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7419.361125] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7424.481617] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7425.984255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7426.300083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7429.738601] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7436.444153] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7440.922535] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7445.927093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7446.261839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7450.209736] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7456.755521] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7464.696112] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 7475.170503] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7475.172245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7475.176332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7475.184683] Lustre: Skipped 1 previous similar message [ 7478.677082] Lustre: server umount lustre-OST0000 complete [ 7482.849243] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7482.855775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7482.866752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7482.871858] Lustre: Skipped 1 previous similar message [ 7485.413763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7485.414437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7487.736972] Lustre: server umount lustre-MDT0000 complete [ 7490.305575] LustreError: 307379:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317677 with bad export cookie 16830610332933089552 [ 7490.317949] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7490.573832] Lustre: server umount lustre-MDT0001 complete [ 7499.077841] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 7500.773634] Key type lgssc unregistered [ 7500.967995] LNet: 311073:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7500.975092] LNet: Removed LNI 192.168.203.153@tcp [ 7501.575861] Key type .llcrypt unregistered [ 7501.582225] Key type ._llcrypt unregistered [ 7509.313624] alg: No test for adler32 (adler32-zlib) [ 7510.069334] Key type ._llcrypt registered [ 7510.071401] Key type .llcrypt registered [ 7510.160315] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7522.336925] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 7531.862888] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7532.519923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7532.830938] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 7532.964179] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 7532.971166] LNet: Accept secure, port 988 [ 7534.615120] Key type lgssc registered [ 7535.377985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7541.281346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7545.773708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7550.140264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7555.119576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7560.495560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7560.541924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7561.724915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7561.751895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7561.811874] Lustre: lustre-MDT0000: new disk, initializing [ 7561.918854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.931455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7564.953330] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7573.741031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7573.852069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.003501] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7574.061903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7574.065184] Lustre: Skipped 1 previous similar message [ 7574.139158] Lustre: lustre-MDT0001: new disk, initializing [ 7574.280690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7574.320459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7574.329786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7578.329855] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7585.534909] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7590.195314] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7596.385403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7596.443840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7596.449128] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7596.686878] Lustre: lustre-OST0000: new disk, initializing [ 7596.693829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7596.810895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7600.631575] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7606.825525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7606.831112] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7608.473237] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7611.781105] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7611.948557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7614.794239] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7614.961579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7616.996917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7617.000494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7617.005565] Lustre: Skipped 1 previous similar message [ 7617.010504] Lustre: Skipped 1 previous similar message [ 7622.113598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7622.119932] Lustre: Skipped 1 previous similar message [ 7622.504346] Lustre: server umount lustre-OST0000 complete [ 7625.697419] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7625.703260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7625.728231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7627.234203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.623934] Lustre: server umount lustre-MDT0000 complete [ 7632.357032] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7632.376106] LustreError: Skipped 1 previous similar message [ 7634.672505] LustreError: 314817:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317822 with bad export cookie 17376918303553716127 [ 7634.675520] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7634.681790] LustreError: 314817:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7634.989427] Lustre: server umount lustre-MDT0001 complete [ 7641.448278] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:57:08 (1761317828) [ 7646.388187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7646.900068] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7646.963383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7650.362797] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7652.321760] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7656.124470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7656.365624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7659.025242] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7664.053968] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7668.165193] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7672.919738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7673.121757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.406768] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7683.302134] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7696.864840] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7696.869071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7696.883759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7696.912356] Lustre: Skipped 1 previous similar message [ 7696.926051] Lustre: Skipped 3 previous similar messages [ 7700.640628] Lustre: server umount lustre-OST0000 complete [ 7707.107508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7707.117989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7707.120875] Lustre: Skipped 1 previous similar message [ 7709.606383] Lustre: server umount lustre-MDT0000 complete [ 7712.110598] LustreError: 319355:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317899 with bad export cookie 17376918303553716981 [ 7712.112698] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7712.116614] LustreError: 319355:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7719.829454] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 7721.494862] Key type lgssc unregistered [ 7721.634705] LNet: 323029:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7721.639913] LNet: Removed LNI 192.168.203.153@tcp [ 7722.074963] Key type .llcrypt unregistered [ 7722.077232] Key type ._llcrypt unregistered [ 7738.391592] alg: No test for adler32 (adler32-zlib) [ 7739.143419] Key type ._llcrypt registered [ 7739.145434] Key type .llcrypt registered [ 7739.219293] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 7748.072991] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7748.685678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7748.931421] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 7749.073705] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 7749.079991] LNet: Accept secure, port 988 [ 7750.769892] Key type lgssc registered [ 7751.487210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7757.590068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7762.043458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7765.881986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7769.846314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7772.775348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7775.852498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7779.918430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7779.970739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7781.188316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7781.216341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7781.268097] Lustre: lustre-MDT0000: new disk, initializing [ 7781.428793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7781.461113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7784.535296] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7792.152409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7792.222066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7792.289450] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7792.329299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7792.341929] Lustre: Skipped 1 previous similar message [ 7792.433714] Lustre: lustre-MDT0001: new disk, initializing [ 7792.507500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7792.546051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7792.554705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7795.776845] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7803.209295] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7807.175274] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7811.733401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7811.788532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7811.792847] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7811.973594] Lustre: lustre-OST2710: new disk, initializing [ 7811.976812] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 7812.018559] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7815.174867] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7817.242699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7817.247483] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7821.976138] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7827.732216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7827.779981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7827.788369] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7827.921382] Lustre: lustre-OST03e8: new disk, initializing [ 7827.924980] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 7827.986748] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7831.554693] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7835.716718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7835.725804] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7838.699243] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7843.328553] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 7843.463802] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7846.164820] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 7846.328129] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7850.058826] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 7850.239971] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7853.003958] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 7853.172128] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7856.608539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7856.612929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7856.624622] Lustre: Skipped 1 previous similar message [ 7861.733623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7861.736966] Lustre: Skipped 6 previous similar messages [ 7862.270923] Lustre: server umount lustre-MDT0000 complete [ 7864.723099] LustreError: 329065:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318052 with bad export cookie 11784926231557906769 [ 7864.725854] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7864.734520] LustreError: 329065:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7864.954979] Lustre: server umount lustre-MDT0001 complete [ 7872.991151] Lustre: 332461:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318054/real 1761318054] req@000000006fb2dd02 x1846875716086848/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761318060 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7873.061649] Lustre: server umount lustre-OST2710 complete [ 7873.887105] Lustre: 324771:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318054/real 1761318054] req@000000006b7b4f18 x1846875716086720/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761318061 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7873.900144] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7873.905444] Lustre: Skipped 2 previous similar messages [ 7876.387705] Lustre: server umount lustre-OST03e8 complete [ 7887.713477] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 7893.422129] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7899.050360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7902.299761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7905.670104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7909.239065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7913.144445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7913.201580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7913.354864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7913.374339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7913.423402] Lustre: lustre-MDT0000: new disk, initializing [ 7913.498471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7913.510834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7916.066559] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7923.829498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7923.904921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7923.984615] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7924.023978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7924.027057] Lustre: Skipped 1 previous similar message [ 7924.134443] Lustre: lustre-MDT0001: new disk, initializing [ 7924.246414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7924.294055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7924.309497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7927.715889] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7935.096191] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7939.598842] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7945.416513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7945.459715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7945.468851] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7945.643379] Lustre: lustre-OST0000: new disk, initializing [ 7945.656825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7945.729094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7949.260756] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 7954.474030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7954.484546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7957.856742] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7960.963666] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7961.118520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7963.840845] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7963.969054] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7969.762772] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7969.764118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7969.774730] Lustre: Skipped 1 previous similar message [ 7969.778187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7969.782281] Lustre: Skipped 1 previous similar message [ 7971.466966] Lustre: server umount lustre-OST0000 complete [ 7974.880390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7974.892683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7974.895293] Lustre: Skipped 1 previous similar message [ 7980.002216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7980.005095] Lustre: Skipped 1 previous similar message [ 7980.448684] Lustre: server umount lustre-MDT0000 complete [ 7983.010675] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7983.021129] LustreError: Skipped 1 previous similar message [ 7983.031162] LustreError: 335573:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318170 with bad export cookie 11784926231557908379 [ 7983.033684] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7983.036054] LustreError: 335573:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7987.859952] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:02:55 (1761318175) [ 7988.761048] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7989.608505] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:02:57 (1761318177) [ 7991.614370] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7997.007464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8000.274625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8002.962703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8005.360679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8005.972955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8010.355403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.572181] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8010.590339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8010.698516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8013.837542] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8019.761472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8019.923082] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8019.946187] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8023.280962] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8029.416414] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8033.887695] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8039.138786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8039.229235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8039.450147] Lustre: Denying initial registration attempt from nid 192.168.203.153@tcp, specified as failover [ 8039.459814] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8039.475950] LustreError: 343590:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 8039.493527] LustreError: 343590:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 8039.496338] LustreError: 343590:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 8039.645463] Lustre: server umount lustre-OST0000 complete [ 8039.651228] Lustre: Skipped 1 previous similar message [ 8039.658643] LustreError: 343590:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8045.024524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.029645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.038316] Lustre: Skipped 2 previous similar messages [ 8045.044702] Lustre: Skipped 1 previous similar message [ 8050.151518] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.165136] LustreError: Skipped 1 previous similar message [ 8051.498821] LustreError: 343862:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318238 with bad export cookie 11784926231557909233 [ 8051.501203] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8051.505896] LustreError: 343862:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8059.043226] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 8060.522131] Key type lgssc unregistered [ 8060.649812] LNet: 344665:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8060.655045] LNet: Removed LNI 192.168.203.153@tcp [ 8061.121348] Key type .llcrypt unregistered [ 8061.122980] Key type ._llcrypt unregistered [ 8070.223500] alg: No test for adler32 (adler32-zlib) [ 8070.982469] Key type ._llcrypt registered [ 8070.988859] Key type .llcrypt registered [ 8071.104101] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8072.157373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8072.469672] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 8072.712545] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 8072.716450] LNet: Accept secure, port 988 [ 8074.367119] Key type lgssc registered [ 8075.379427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8081.753682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8085.027478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8088.386247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8092.273659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8093.350843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8104.879108] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8111.717402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8112.919755] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8112.942292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8113.108280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8116.282884] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8122.694261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.766436] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8122.795472] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8122.977675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8127.300501] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8133.115850] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8137.141085] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8142.188755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8142.250554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8142.394805] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8142.544039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8145.475684] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8150.946764] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8153.060654] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8153.061716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8153.064526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8153.072678] Lustre: Skipped 1 previous similar message [ 8154.362716] Lustre: server umount lustre-OST0000 complete [ 8158.183612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8158.185470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8158.193026] Lustre: Skipped 1 previous similar message [ 8158.202431] Lustre: Skipped 2 previous similar messages [ 8163.136613] Lustre: server umount lustre-MDT0000 complete [ 8163.296362] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8163.313531] LustreError: Skipped 1 previous similar message [ 8166.004398] LustreError: 349520:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318353 with bad export cookie 14037807769627467848 [ 8166.005927] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8166.012445] LustreError: 349520:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8166.200593] Lustre: server umount lustre-MDT0001 complete [ 8175.044639] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 8177.358124] Key type lgssc unregistered [ 8177.564586] LNet: 351236:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8177.574425] LNet: Removed LNI 192.168.203.153@tcp [ 8178.234821] Key type .llcrypt unregistered [ 8178.246901] Key type ._llcrypt unregistered [ 8195.952669] alg: No test for adler32 (adler32-zlib) [ 8196.710686] Key type ._llcrypt registered [ 8196.712909] Key type .llcrypt registered [ 8196.792395] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8197.532685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8197.910299] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 8198.073282] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 8198.077691] LNet: Accept secure, port 988 [ 8199.786994] Key type lgssc registered [ 8200.730792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8207.552626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8208.960969] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8209.013364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8212.472541] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8214.497468] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.493342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.618308] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8219.739119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8222.991553] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8228.800680] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8233.164859] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8239.062542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8239.544382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8244.424242] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8252.951731] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8269.026353] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 8269.282447] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8270.937827] Lustre: server umount lustre-MDT0000 complete [ 8273.925663] LustreError: 352677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318461 with bad export cookie 763845382168909376 [ 8273.929108] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8274.247805] Lustre: server umount lustre-MDT0001 complete [ 8279.743748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8287.097534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8290.794306] Lustre: Evicted from MGS (at 192.168.203.153@tcp) after server handle changed from 0xa99b953fb801640 to 0xa99b953fb80322b [ 8290.812871] Lustre: MGC192.168.203.153@tcp: Connection restored to 192.168.203.153@tcp (at 0@lo) [ 8291.028079] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8291.035250] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8291.050057] Lustre: Skipped 2 previous similar messages [ 8291.115404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8291.155721] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 8294.888523] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8295.678781] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8295.683448] LustreError: Skipped 1 previous similar message [ 8300.755077] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8300.777706] LustreError: Skipped 2 previous similar messages [ 8302.534625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8302.573204] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 8302.928815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8308.146705] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8308.219717] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 8316.429663] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8318.216402] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8318.372925] Lustre: 352008:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318461/real 1761318461] req@000000008bdefa49 x1846876186884032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761318505 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8323.010973] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8323.039140] Lustre: 352008:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318465/real 1761318465] req@00000000f131ec3f x1846876186884096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761318509 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8324.178724] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8324.576291] Lustre: 352010:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318467/real 1761318467] req@000000001330760b x1846876186884288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761318511 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8324.605916] Lustre: 352010:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8328.677686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8328.677983] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8328.690585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8328.708142] Lustre: Skipped 4 previous similar messages [ 8334.015763] Lustre: server umount lustre-OST0000 complete [ 8338.913682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8338.917081] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8338.918544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8338.930339] Lustre: Skipped 2 previous similar messages [ 8343.494480] Lustre: server umount lustre-MDT0000 complete [ 8344.035527] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8344.070268] LustreError: Skipped 6 previous similar messages [ 8346.827869] LustreError: 352677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318534 with bad export cookie 763845382168916523 [ 8346.839602] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8346.861949] LustreError: 352677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8359.336267] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 8362.696331] Key type lgssc unregistered [ 8363.080579] LNet: 359216:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8363.086937] LNet: Removed LNI 192.168.203.153@tcp [ 8363.942251] Key type .llcrypt unregistered [ 8363.948636] Key type ._llcrypt unregistered [ 8382.797042] alg: No test for adler32 (adler32-zlib) [ 8383.547339] Key type ._llcrypt registered [ 8383.549475] Key type .llcrypt registered [ 8383.637984] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8384.609846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8384.993877] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 8385.156689] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 8385.160701] LNet: Accept secure, port 988 [ 8386.833083] Key type lgssc registered [ 8387.636911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8394.698253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8396.226346] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8396.274839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8400.203917] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8401.376216] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8406.499235] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8406.895365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.232914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8410.423550] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8416.483398] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8422.029850] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8428.003725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8428.003852] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8428.016163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8428.052451] Lustre: Skipped 1 previous similar message [ 8433.121164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8433.123992] Lustre: Skipped 1 previous similar message [ 8437.216963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8437.225663] Lustre: Skipped 1 previous similar message [ 8439.263303] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8439.362906] Lustre: server umount lustre-MDT0000 complete [ 8442.276628] LustreError: 361465:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318629 with bad export cookie 16647223121379571628 [ 8442.282346] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8442.290200] LustreError: 361465:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8442.601523] Lustre: server umount lustre-MDT0001 complete [ 8453.322101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8453.702699] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8453.738166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8453.897130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8457.588234] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8464.537447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8464.593261] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8464.613084] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8464.769429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8467.620930] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8473.364748] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8477.994135] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8483.671660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8483.818525] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8483.826398] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8483.830450] LustreError: 365372:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 8483.834805] LustreError: 365372:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 8483.840857] LustreError: 365372:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 8483.934673] Lustre: server umount lustre-OST0000 complete [ 8483.936466] LustreError: 365372:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8488.459732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8488.613914] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8488.729889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8492.091242] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8495.728055] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 8498.687869] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8505.109449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8505.167834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8505.175802] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 8505.231303] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8505.292967] Lustre: lustre-OST0001: new disk, initializing [ 8505.295979] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 8505.370589] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8508.755699] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8513.619608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8513.633060] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8514.881840] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8519.137786] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8519.146525] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8519.163798] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8519.363205] Lustre: server umount lustre-OST0001 complete [ 8524.259203] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8524.259728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8524.272044] LustreError: Skipped 1 previous similar message [ 8524.276603] Lustre: Skipped 1 previous similar message [ 8524.285225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8524.291765] Lustre: Skipped 2 previous similar messages [ 8528.854882] Lustre: server umount lustre-OST0000 complete [ 8534.498403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8534.500116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8534.526026] Lustre: Skipped 2 previous similar messages [ 8546.271235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8546.350544] Lustre: server umount lustre-MDT0000 complete [ 8549.122735] LustreError: 363372:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318736 with bad export cookie 16647223121379572181 [ 8549.124764] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8549.128196] LustreError: 363372:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8557.599352] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 8559.833417] Key type lgssc unregistered [ 8560.077509] LNet: 368742:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8560.095596] LNet: Removed LNI 192.168.203.153@tcp [ 8560.808222] Key type .llcrypt unregistered [ 8560.810406] Key type ._llcrypt unregistered [ 8585.605337] alg: No test for adler32 (adler32-zlib) [ 8586.359440] Key type ._llcrypt registered [ 8586.362129] Key type .llcrypt registered [ 8586.439743] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 8596.107384] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8596.771554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8596.973507] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 8597.123839] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 8597.125994] LNet: Accept secure, port 988 [ 8598.767205] Key type lgssc registered [ 8599.546702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8606.113043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8611.005369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8615.273948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8620.100935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8625.344742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8625.395435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8626.572396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8626.613194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8626.698567] Lustre: lustre-MDT0000: new disk, initializing [ 8626.818445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8626.832964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8630.123753] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8638.829166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8638.892811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8638.949934] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8638.985156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8638.988024] Lustre: Skipped 1 previous similar message [ 8639.035859] Lustre: lustre-MDT0001: new disk, initializing [ 8639.106076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8639.134206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8639.145882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8642.379407] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8649.388096] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8653.484898] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8658.044472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8658.100243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8658.105171] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8658.294703] Lustre: lustre-OST0000: new disk, initializing [ 8658.300554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8658.358686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8661.008730] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8667.051359] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8667.694703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8667.699247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8669.829444] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8669.977368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8672.868910] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8673.045346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8677.861296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8677.864781] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8677.877126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8677.884757] Lustre: Skipped 1 previous similar message [ 8680.488451] Lustre: server umount lustre-OST0000 complete [ 8688.096846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.105802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8688.108663] Lustre: Skipped 2 previous similar messages [ 8688.115979] Lustre: Skipped 1 previous similar message [ 8689.337557] Lustre: server umount lustre-MDT0000 complete [ 8692.084067] LustreError: 372830:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318879 with bad export cookie 11179996519093587357 [ 8692.088048] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8692.090956] LustreError: 372830:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8692.284298] Lustre: server umount lustre-MDT0001 complete [ 8697.744472] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 11:14:45 (1761318885) [ 8701.337639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8705.367637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8721.991969] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 8727.567171] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8732.916950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8736.534587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8739.964340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8743.851899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8748.520444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8748.570576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.737474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8748.778838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8748.848502] Lustre: lustre-MDT0000: new disk, initializing [ 8748.934704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8748.948412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8751.729589] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8758.914113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8758.968525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8759.022707] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8759.056908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8759.063760] Lustre: Skipped 1 previous similar message [ 8759.110878] Lustre: lustre-MDT0001: new disk, initializing [ 8759.171308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8759.204073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8759.209849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8762.018444] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8768.653934] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8772.325341] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8775.672806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8775.709891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8775.715139] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8775.843799] Lustre: lustre-OST0000: new disk, initializing [ 8775.849564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8775.898171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8777.994937] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8783.681319] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8784.906387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8784.912355] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8785.934025] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8786.044263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8787.946050] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8788.057534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8789.985802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8789.989271] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8790.001135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8790.007561] Lustre: Skipped 1 previous similar message [ 8795.072295] Lustre: server umount lustre-OST0000 complete [ 8800.225504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8800.226190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8800.226992] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8800.233031] Lustre: Skipped 2 previous similar messages [ 8800.240819] Lustre: Skipped 1 previous similar message [ 8803.217245] Lustre: server umount lustre-MDT0000 complete [ 8804.984413] LustreError: 380596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318992 with bad export cookie 11179996519093588211 [ 8804.988319] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8804.990247] LustreError: 380596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8809.911720] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 11:16:37 (1761318997) [ 8814.888846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8815.176985] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8815.241655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8818.257233] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8820.708666] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8824.780481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8828.106742] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8833.341755] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8836.827439] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8840.764797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8843.530108] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8848.457951] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8863.425517] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 8864.477419] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 8865.427563] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 8866.360893] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 8867.349724] Lustre: Failing over lustre-MDT0000 [ 8867.512534] Lustre: server umount lustre-MDT0000 complete [ 8867.515275] Lustre: Skipped 1 previous similar message [ 8870.622195] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8870.881935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8870.888151] Lustre: Skipped 2 previous similar messages [ 8875.733810] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8875.742682] LustreError: Skipped 3 previous similar messages [ 8878.047249] Lustre: 371058:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319058/real 1761319058] req@00000000acca6229 x1846876605296896/t0(0) o400->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761319065 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8878.058105] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8882.801973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8882.893522] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8882.905332] LustreError: Skipped 7 previous similar messages [ 8884.194961] Lustre: Evicted from MGS (at 192.168.203.153@tcp) after server handle changed from 0x9b27539996ad5450 to 0x9b27539996ad5839 [ 8884.212451] Lustre: MGC192.168.203.153@tcp: Connection restored to 192.168.203.153@tcp (at 0@lo) [ 8884.362143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8884.364697] Lustre: Skipped 2 previous similar messages [ 8884.402650] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8886.916801] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8888.015592] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8889.835703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 8889.847824] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8889.877680] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 8893.143317] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8894.125840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8896.448377] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 8897.401231] Lustre: Failing over lustre-MDT0000 [ 8897.550107] Lustre: server umount lustre-MDT0000 complete [ 8898.260694] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8898.266360] LustreError: Skipped 6 previous similar messages [ 8900.067330] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.069137] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8900.072855] Lustre: Skipped 1 previous similar message [ 8906.655138] Lustre: 371060:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319087/real 1761319087] req@00000000c719ef94 x1846876605305024/t0(0) o400->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761319094 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8906.672120] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8912.243779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8912.869348] Lustre: Evicted from MGS (at 192.168.203.153@tcp) after server handle changed from 0x9b27539996ad5839 to 0x9b27539996ad5b3b [ 8912.875122] Lustre: MGC192.168.203.153@tcp: Connection restored to (at 0@lo) [ 8912.877604] Lustre: Skipped 2 previous similar messages [ 8913.012315] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8915.247830] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8916.686600] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8917.988293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 8918.000674] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8918.031242] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 8921.341219] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8922.251874] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8928.225619] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8928.225624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.225631] Lustre: Skipped 1 previous similar message [ 8928.229804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8928.241358] Lustre: Skipped 2 previous similar messages [ 8931.801511] Lustre: server umount lustre-MDT0000 complete [ 8933.349139] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8933.355206] LustreError: Skipped 21 previous similar messages [ 8933.552371] LustreError: 385232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319120 with bad export cookie 11179996519093590843 [ 8933.554194] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8933.557211] LustreError: 385232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8941.537223] Lustre: 391054:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319122/real 1761319122] req@000000004b8246c9 x1846876605314688/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761319128 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8949.891883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8950.179829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8950.184108] Lustre: Skipped 1 previous similar message [ 8952.156194] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8956.072697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8958.440550] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8962.443530] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8965.550941] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8969.118631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8971.394180] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 8975.289893] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8980.582847] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 8986.080930] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8986.082316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8986.085717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8986.092795] Lustre: Skipped 2 previous similar messages [ 8986.096886] Lustre: Skipped 1 previous similar message [ 8997.082589] Lustre: server umount lustre-MDT0000 complete [ 8997.084890] Lustre: Skipped 3 previous similar messages [ 8999.030408] LustreError: 391863:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319186 with bad export cookie 11179996519093591543 [ 8999.032520] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8999.041385] LustreError: 391863:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9006.008054] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 9007.549835] Key type lgssc unregistered [ 9007.721314] LNet: 395533:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9007.725866] LNet: Removed LNI 192.168.203.153@tcp [ 9008.112397] Key type .llcrypt unregistered [ 9008.114076] Key type ._llcrypt unregistered [ 9022.614364] alg: No test for adler32 (adler32-zlib) [ 9023.369557] Key type ._llcrypt registered [ 9023.371262] Key type .llcrypt registered [ 9023.420619] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9023.829644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9024.076713] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 9024.178688] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 9024.181364] LNet: Accept secure, port 988 [ 9025.783518] Key type lgssc registered [ 9026.344615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9030.581094] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9030.588747] LustreError: 397016:0:(osd_handler.c:8211:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9030.593608] LustreError: 397016:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9030.597805] LustreError: 397016:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9030.602717] LustreError: 397016:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9030.608180] LustreError: 397016:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9035.567944] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9035.574145] LustreError: 397413:0:(osd_handler.c:8211:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9035.577378] LustreError: 397413:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9035.581239] LustreError: 397413:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9035.586876] LustreError: 397413:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9035.590780] LustreError: 397413:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9044.242603] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 9045.852979] Key type lgssc unregistered [ 9046.008308] LNet: 398370:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9046.015884] LNet: Removed LNI 192.168.203.153@tcp [ 9046.515448] Key type .llcrypt unregistered [ 9046.517408] Key type ._llcrypt unregistered [ 9057.960176] alg: No test for adler32 (adler32-zlib) [ 9058.710420] Key type ._llcrypt registered [ 9058.713456] Key type .llcrypt registered [ 9058.804635] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 9068.005556] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9068.528443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9068.904315] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 9069.043159] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 9069.046408] LNet: Accept secure, port 988 [ 9070.655149] Key type lgssc registered [ 9071.261229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9076.396584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9079.707132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9082.851711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9085.851295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9088.936178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9088.969979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9090.080898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9090.095936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9090.144749] Lustre: lustre-MDT0000: new disk, initializing [ 9090.194380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9090.205338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9092.476552] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9099.120143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9099.171797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9099.238083] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9099.269639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9099.272526] Lustre: Skipped 1 previous similar message [ 9099.327626] Lustre: lustre-MDT0001: new disk, initializing [ 9099.391744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9099.418316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9099.424306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9101.841580] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9107.451954] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9111.033930] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9114.963267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9114.999865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9115.004939] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9115.173965] Lustre: lustre-OST0000: new disk, initializing [ 9115.177133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9115.230534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9118.046726] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9123.338986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9123.343219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9123.417891] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9125.596736] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9125.708504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9127.615927] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9127.717412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9130.785292] Lustre: server umount lustre-OST0000 complete [ 9133.536629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9133.538496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9133.540789] Lustre: Skipped 1 previous similar message [ 9138.655841] Lustre: 405322:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319319/real 1761319319] req@00000000a3ecc1fc x1846877100193856/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761319325 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9138.656311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9138.667546] Lustre: Skipped 2 previous similar messages [ 9138.734867] Lustre: server umount lustre-MDT0000 complete [ 9139.679114] Lustre: 399870:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319320/real 1761319320] req@00000000118bb86b x1846877100194240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761319327 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9139.687822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9139.693766] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9139.699942] LustreError: Skipped 1 previous similar message [ 9140.344373] LustreError: 403727:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319327 with bad export cookie 12292213529237094472 [ 9140.347217] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9140.349343] LustreError: 403727:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9140.489541] Lustre: server umount lustre-MDT0001 complete [ 9144.545171] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9145.369563] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:22:12 (1761319332) [ 9148.434894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9148.635742] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9148.667786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9150.482974] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9154.020628] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9154.480361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9154.665653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9156.763222] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9160.209224] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9162.892138] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9165.999573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9166.172908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9168.226158] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9171.845036] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9174.722512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9174.757897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9174.763303] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 9174.827819] Lustre: lustre-OST0001: new disk, initializing [ 9174.829735] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 9176.574146] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9178.249036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9178.253871] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9180.981917] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9183.712554] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9183.716304] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.722312] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9183.724764] Lustre: Skipped 1 previous similar message [ 9185.072882] Lustre: server umount lustre-OST0001 complete [ 9186.014211] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9194.981971] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9194.989907] LustreError: Skipped 3 previous similar messages [ 9200.096792] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9200.099222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9200.100409] LustreError: Skipped 1 previous similar message [ 9200.101045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9200.106193] Lustre: Skipped 2 previous similar messages [ 9200.108212] Lustre: Skipped 2 previous similar messages [ 9205.219922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9205.220358] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9205.222878] Lustre: Skipped 1 previous similar message [ 9205.228586] LustreError: Skipped 2 previous similar messages [ 9212.383136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9212.458128] Lustre: server umount lustre-OST0000 complete [ 9215.458137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9215.463875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9215.467039] Lustre: Skipped 3 previous similar messages [ 9222.003465] LustreError: 409143:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319409 with bad export cookie 12292213529237095326 [ 9222.006447] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9222.010119] LustreError: 409143:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9228.260718] Lustre: server umount lustre-MDT0001 complete [ 9228.262679] Lustre: Skipped 1 previous similar message [ 9233.722401] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 9234.848648] Key type lgssc unregistered [ 9234.971362] LNet: 411141:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9234.974734] LNet: Removed LNI 192.168.203.153@tcp [ 9235.289234] Key type .llcrypt unregistered [ 9235.290663] Key type ._llcrypt unregistered [ 9248.401236] alg: No test for adler32 (adler32-zlib) [ 9249.153392] Key type ._llcrypt registered [ 9249.154566] Key type .llcrypt registered [ 9249.210458] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 9256.405025] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9256.824785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9257.047345] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 9257.139882] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 9257.141922] LNet: Accept secure, port 988 [ 9258.735087] Key type lgssc registered [ 9259.204195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9263.328208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9266.984329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9269.948752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9273.317458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9276.285190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9276.321857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9277.418592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9277.432607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9277.476642] Lustre: lustre-MDT0000: new disk, initializing [ 9277.513541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9277.522122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9279.253185] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9284.016642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9284.044619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9284.093539] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9284.120692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9284.123087] Lustre: Skipped 1 previous similar message [ 9284.161568] Lustre: lustre-MDT0001: new disk, initializing [ 9284.201586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9284.216388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9284.221621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9285.936855] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9290.462228] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9292.785837] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9295.479807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9295.516566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9295.520694] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9295.626289] Lustre: lustre-OST0000: new disk, initializing [ 9295.628840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9295.660404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9297.447403] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9301.967688] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9302.013238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9302.017625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9303.891995] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9303.989267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9305.738454] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9305.826574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9307.105173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9307.105894] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9307.106219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9307.109776] Lustre: Skipped 1 previous similar message [ 9308.638803] Lustre: server umount lustre-OST0000 complete [ 9312.226160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9312.226919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9312.231549] Lustre: Skipped 1 previous similar message [ 9312.234015] Lustre: Skipped 1 previous similar message [ 9316.168409] Lustre: server umount lustre-MDT0000 complete [ 9317.343743] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9317.349304] LustreError: Skipped 1 previous similar message [ 9317.677342] LustreError: 415223:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319505 with bad export cookie 11993113109361036821 [ 9317.678158] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9317.681517] LustreError: 415223:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9317.820246] Lustre: server umount lustre-MDT0001 complete [ 9321.340377] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 11:25:08 (1761319508) [ 9324.892148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9328.200784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9328.347511] Lustre: lustre-MDT0000: new disk, initializing [ 9328.359426] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9328.381708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9329.953648] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9333.362949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9333.488472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9335.151482] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9338.157441] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9340.505518] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9343.969073] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9343.969519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9343.971565] LustreError: Skipped 1 previous similar message [ 9343.972161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9343.975803] Lustre: Skipped 1 previous similar message [ 9343.982753] Lustre: Skipped 2 previous similar messages [ 9347.464288] Lustre: server umount lustre-MDT0000 complete [ 9348.892054] LustreError: 420350:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319536 with bad export cookie 11993113109361037661 [ 9348.894387] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9348.896343] LustreError: 420350:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9352.835703] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:25:40 (1761319540) [ 9355.578740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9355.783169] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9355.820412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9357.555931] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9360.818806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9360.864759] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9362.656461] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9365.787424] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9367.961210] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9370.400536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9372.105315] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9374.810394] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9382.329496] Lustre: Permanently deactivating lustre-OST0000 [ 9382.336414] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 9385.263314] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9385.611533] LustreError: 425911:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 9385.616654] LustreError: 425911:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 9385.619545] LustreError: 425911:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 9386.312539] LustreError: 425959:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 9386.315933] LustreError: 425959:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 9393.191887] Lustre: server umount lustre-OST0000 complete [ 9393.194380] Lustre: Skipped 1 previous similar message [ 9394.425527] LustreError: 426209:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 9394.428783] LustreError: 426209:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 9396.704119] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9396.707574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9396.713236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9401.312717] Lustre: 426306:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319582/real 1761319582] req@0000000033d98e7b x1846877297346176/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761319588 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9401.464873] Lustre: server umount lustre-MDT0000 complete [ 9402.338389] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9402.351871] LustreError: Skipped 1 previous similar message [ 9403.748107] LustreError: 422945:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319591 with bad export cookie 11993113109361038200 [ 9403.748740] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9403.762628] LustreError: 422945:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9410.015656] Lustre: 426507:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319591/real 1761319591] req@00000000bafb9dbb x1846877297347264/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761319597 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9421.036034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9426.123774] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9429.080983] LustreError: 427417:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 9436.169920] Lustre: 427763:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.153@tcp [ 9443.807172] Lustre: 427862:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319625/real 1761319625] req@00000000594d1bfc x1846877297347776/t0(0) o251->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761319631 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9443.865819] Lustre: server umount MGS complete [ 9443.866946] Lustre: Skipped 1 previous similar message [ 9457.516117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9458.361587] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9458.405629] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9458.418163] Lustre: Skipped 1 previous similar message [ 9458.523317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9458.530442] Lustre: Skipped 2 previous similar messages [ 9463.984590] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9471.905747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9472.213824] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9472.216707] LustreError: 413453:0:(client.c:1266:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009879e2a6 x1846877297350848/t0(0) o8->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/0/ffffffff rc 0/-1 job:'' [ 9476.462988] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9483.176310] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9487.653304] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9489.130567] Lustre: Permanently reactivating lustre-OST0000 [ 9489.153110] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 9489.156594] Lustre: Skipped 2 previous similar messages [ 9510.109803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9510.426836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9510.431385] Lustre: Skipped 1 previous similar message [ 9514.081834] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9520.345440] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9524.704860] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9524.709649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9524.739396] Lustre: Skipped 1 previous similar message [ 9524.751357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9524.755231] Lustre: Skipped 3 previous similar messages [ 9526.310435] Lustre: server umount lustre-OST0000 complete [ 9534.944345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9534.952623] Lustre: Skipped 3 previous similar messages [ 9538.895337] LustreError: 428473:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319726 with bad export cookie 11993113109361039145 [ 9538.901201] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9538.902635] LustreError: 428473:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9548.212481] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 9550.341277] Key type lgssc unregistered [ 9550.591139] LNet: 432736:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9550.598100] LNet: Removed LNI 192.168.203.153@tcp [ 9551.226178] Key type .llcrypt unregistered [ 9551.229617] Key type ._llcrypt unregistered [ 9564.992429] alg: No test for adler32 (adler32-zlib) [ 9565.744447] Key type ._llcrypt registered [ 9565.745986] Key type .llcrypt registered [ 9565.823563] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 9576.751073] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9577.526363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9577.794195] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 9577.962820] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 9577.968962] LNet: Accept secure, port 988 [ 9579.648132] Key type lgssc registered [ 9580.344207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9586.519761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9592.603215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9598.636081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9603.823307] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9609.595110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9609.642355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9610.815885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9610.838101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9610.894925] Lustre: lustre-MDT0000: new disk, initializing [ 9610.958328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9610.978965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9613.935404] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9622.327237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9622.395071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9622.496669] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9622.531278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9622.534565] Lustre: Skipped 1 previous similar message [ 9622.597533] Lustre: lustre-MDT0001: new disk, initializing [ 9622.749848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9622.822248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9622.836050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9626.381162] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9633.732547] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9638.159321] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9643.187352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9643.238030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9643.241581] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9643.419483] Lustre: lustre-OST0000: new disk, initializing [ 9643.425899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9643.517929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9646.495659] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9650.737677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9650.754514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9654.570293] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9658.656321] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9658.885318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9662.485115] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9662.692380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9666.019176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9666.030883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9666.047820] Lustre: Skipped 1 previous similar message [ 9670.547691] Lustre: server umount lustre-OST0000 complete [ 9674.208947] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9674.221207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9674.232254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9674.236027] Lustre: Skipped 1 previous similar message [ 9676.255874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9676.259723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9676.271734] Lustre: Skipped 1 previous similar message [ 9679.411532] Lustre: server umount lustre-MDT0000 complete [ 9681.377713] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9681.390576] LustreError: Skipped 1 previous similar message [ 9682.401957] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9682.433496] LustreError: Skipped 1 previous similar message [ 9682.733185] LustreError: 438094:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319870 with bad export cookie 15009564121216135689 [ 9682.736396] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9682.749820] LustreError: 438094:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9683.071138] Lustre: server umount lustre-MDT0001 complete [ 9688.980986] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9690.167979] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:31:17 (1761319877) [ 9694.902358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9695.211482] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9695.281415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9698.567873] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9700.323492] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9704.724374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9704.969397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9708.183656] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9714.020356] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9719.151403] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9725.110826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9725.415370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9728.493734] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9733.974490] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9735.481218] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9737.730911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9740.772805] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 9740.774539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9740.775821] LustreError: Skipped 1 previous similar message [ 9740.775879] LustreError: 441387:0:(osp_precreate.c:677:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -107 [ 9740.782666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9740.785115] Lustre: Skipped 1 previous similar message [ 9745.893549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9745.895810] Lustre: Skipped 1 previous similar message [ 9746.906418] Lustre: server umount lustre-OST0000 complete [ 9751.008898] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9751.009389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9751.015671] Lustre: Skipped 1 previous similar message [ 9755.818408] Lustre: server umount lustre-MDT0000 complete [ 9756.130070] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9756.146034] LustreError: Skipped 1 previous similar message [ 9758.231241] LustreError: 440587:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319945 with bad export cookie 15009564121216136543 [ 9758.233291] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9758.238865] LustreError: 440587:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9766.450780] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 9768.644057] Key type lgssc unregistered [ 9768.829519] LNet: 444453:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9768.838870] LNet: Removed LNI 192.168.203.153@tcp [ 9769.385319] Key type .llcrypt unregistered [ 9769.389211] Key type ._llcrypt unregistered [ 9787.044136] alg: No test for adler32 (adler32-zlib) [ 9787.799392] Key type ._llcrypt registered [ 9787.802382] Key type .llcrypt registered [ 9787.882804] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 9788.076106] Key type .llcrypt unregistered [ 9788.078223] Key type ._llcrypt unregistered [ 9800.398660] alg: No test for adler32 (adler32-zlib) [ 9801.156080] Key type ._llcrypt registered [ 9801.157204] Key type .llcrypt registered [ 9801.285844] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9802.063516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9802.375187] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 9802.558660] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 9802.561172] LNet: Accept secure, port 988 [ 9804.199172] Key type lgssc registered [ 9804.889947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9810.646956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9812.009114] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9812.051086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9815.164390] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9817.061980] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9821.570591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9821.840274] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9821.855506] LustreError: Skipped 1 previous similar message [ 9821.869405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9825.243647] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9826.979224] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9826.993179] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 9827.003494] LustreError: Skipped 1 previous similar message [ 9831.111045] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9832.417073] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9832.437375] LustreError: Skipped 1 previous similar message [ 9836.627907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9836.954416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9840.353364] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9847.270125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9847.270588] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9847.271041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9847.281050] Lustre: Skipped 1 previous similar message [ 9849.044508] Lustre: server umount lustre-OST0000 complete [ 9852.388276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9852.391669] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9852.407092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9852.412133] Lustre: Skipped 2 previous similar messages [ 9857.508681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9857.515590] Lustre: Skipped 1 previous similar message [ 9858.140567] Lustre: server umount lustre-MDT0000 complete [ 9861.062758] LustreError: 446764:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320048 with bad export cookie 2165177480137769471 [ 9861.074441] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9861.085379] LustreError: 446764:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9861.342770] Lustre: server umount lustre-MDT0001 complete [ 9870.617361] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 9872.727279] Key type lgssc unregistered [ 9873.003244] LNet: 450043:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9873.012215] LNet: Removed LNI 192.168.203.153@tcp [ 9873.674941] Key type .llcrypt unregistered [ 9873.676809] Key type ._llcrypt unregistered [ 9891.569712] alg: No test for adler32 (adler32-zlib) [ 9892.328446] Key type ._llcrypt registered [ 9892.329961] Key type .llcrypt registered [ 9892.434509] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9893.099154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9893.512673] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 9893.717897] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 9893.721485] LNet: Accept secure, port 988 [ 9895.415125] Key type lgssc registered [ 9896.327511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9902.056855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9919.647281] LustreError: 451477:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.153@tcp: failed processing log, type 1: rc = -5 [ 9950.368686] LustreError: 451477:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.153@tcp: failed processing log, type 4: rc = -110 [ 9980.127181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9982.687226] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [ 9987.346984] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9991.765715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10002.796834] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.893934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10002.983226] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [10006.205796] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10008.033101] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10011.448814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10011.665662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10014.447155] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10019.451647] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10023.729516] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10028.514147] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10028.518643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10028.530354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10030.047893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10030.049227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10034.304437] Lustre: server umount lustre-OST0000 complete [10036.644619] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10036.649270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.675641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10036.678090] Lustre: Skipped 1 previous similar message [10040.289112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10040.291707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10042.810430] Lustre: server umount lustre-MDT0000 complete [10045.376815] LustreError: 451483:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320232 with bad export cookie 1976915114895542917 [10045.378444] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10045.382609] LustreError: 451483:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10045.571859] Lustre: server umount lustre-MDT0001 complete [10054.177796] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [10056.137596] Key type lgssc unregistered [10056.404319] LNet: 455168:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10056.420824] LNet: Removed LNI 192.168.203.153@tcp [10057.066381] Key type .llcrypt unregistered [10057.068715] Key type ._llcrypt unregistered [10073.538470] alg: No test for adler32 (adler32-zlib) [10074.292391] Key type ._llcrypt registered [10074.294091] Key type .llcrypt registered [10074.412767] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10074.962556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10075.267164] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [10075.492344] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [10075.495546] LNet: Accept secure, port 988 [10077.143210] Key type lgssc registered [10077.912144] Lustre: Echo OBD driver; http://www.lustre.org/ [10083.451633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.751089] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10084.821137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10087.758779] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10089.952383] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10092.391788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10092.596427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10095.288359] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10101.167250] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10105.581122] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10110.241735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10110.465436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10113.227648] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10115.630162] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [10119.171517] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10123.232379] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10123.237037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10123.243994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10126.054942] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.53@tcp (stopping) [10126.823120] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10126.837704] Lustre: Skipped 1 previous similar message [10128.519277] Lustre: server umount lustre-MDT0000 complete [10130.423510] LustreError: 458577:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.53@tcp arrived at 1761320317 with bad export cookie 7728717671014824546 [10130.436263] LustreError: 458577:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10131.884269] LustreError: 458577:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320319 with bad export cookie 7728717671014823664 [10131.895575] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10131.925624] LustreError: 458577:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10131.939367] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10131.941775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10131.942777] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10131.963155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10131.965645] LustreError: Skipped 1 previous similar message [10131.969524] Lustre: Skipped 3 previous similar messages [10137.056605] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10137.057039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10137.064892] LustreError: Skipped 1 previous similar message [10138.049300] Lustre: server umount lustre-OST0000 complete [10142.669897] Lustre: server umount lustre-MDT0001 complete [10150.807503] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [10153.141512] Key type lgssc unregistered [10153.371392] LNet: 460432:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10153.380748] LNet: Removed LNI 192.168.203.153@tcp [10154.033635] Key type .llcrypt unregistered [10154.035409] Key type ._llcrypt unregistered [10170.900536] alg: No test for adler32 (adler32-zlib) [10171.654418] Key type ._llcrypt registered [10171.658180] Key type .llcrypt registered [10171.778901] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10172.552864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10172.828483] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [10173.019459] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [10173.022465] LNet: Accept secure, port 988 [10174.679111] Key type lgssc registered [10175.512510] Lustre: Echo OBD driver; http://www.lustre.org/ [10180.994133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10182.292739] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10182.333717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10185.712113] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10187.744476] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.853948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10192.068158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10195.019346] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10200.270810] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10204.761356] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10209.629159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10209.928298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10212.862239] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10212.974926] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [10217.637647] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10220.135992] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10220.149564] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10220.152528] Lustre: Skipped 1 previous similar message [10224.336392] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.53@tcp (stopping) [10225.632677] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10226.316631] Lustre: server umount lustre-MDT0001 complete [10230.751905] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10230.752563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10230.771356] Lustre: Skipped 1 previous similar message [10230.775412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10230.778146] Lustre: Skipped 1 previous similar message [10235.873308] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10235.873986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10236.348272] Lustre: server umount lustre-OST0000 complete [10245.087180] Lustre: 464943:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320426/real 1761320426] req@0000000050cf6854 x1846878257820608/t0(0) o251->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761320432 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10245.215909] Lustre: server umount lustre-MDT0000 complete [10254.839208] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [10256.513928] Key type lgssc unregistered [10256.704512] LNet: 465697:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10256.710318] LNet: Removed LNI 192.168.203.153@tcp [10257.318559] Key type .llcrypt unregistered [10257.321240] Key type ._llcrypt unregistered [10275.171839] alg: No test for adler32 (adler32-zlib) [10275.928551] Key type ._llcrypt registered [10275.930499] Key type .llcrypt registered [10276.019165] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [10284.214575] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10284.892871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10285.136987] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [10285.267497] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [10285.272114] LNet: Accept secure, port 988 [10286.887308] Key type lgssc registered [10287.568896] Lustre: Echo OBD driver; http://www.lustre.org/ [10293.169362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10297.036641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10300.656586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10304.433509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10310.037700] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10316.068544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10320.907753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10320.954390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10322.131773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10322.157504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10322.215593] Lustre: lustre-MDT0000: new disk, initializing [10322.283046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10322.301517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10325.406218] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10331.905559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10336.725887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10336.794986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10336.799135] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10336.943803] Lustre: lustre-OST0000: new disk, initializing [10336.947798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10336.951268] Lustre: Skipped 1 previous similar message [10337.015173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10339.683430] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10345.861360] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10351.487847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10355.517965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10355.593787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10355.659157] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10355.712566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10355.840366] Lustre: lustre-MDT0001: new disk, initializing [10355.902259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10355.918462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10355.971467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10359.374912] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10365.453623] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [10365.607726] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10367.895644] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [10368.011279] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10373.601346] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10373.601839] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10373.611281] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10373.619741] Lustre: Skipped 1 previous similar message [10373.624486] Lustre: Skipped 1 previous similar message [10374.879674] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.53@tcp (stopping) [10376.597265] Lustre: server umount lustre-MDT0001 complete [10378.722671] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10378.726867] LustreError: Skipped 1 previous similar message [10383.841580] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10383.849632] LustreError: Skipped 1 previous similar message [10388.963149] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.969607] LustreError: Skipped 1 previous similar message [10394.081127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10394.081156] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10394.081258] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10394.087395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10399.201950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10399.202264] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10404.319148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10404.320964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10404.390863] Lustre: server umount lustre-OST0000 complete [10418.655226] Lustre: 474098:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320599/real 1761320599] req@000000000906861a x1846878375265344/t0(0) o251->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761320605 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10418.750045] Lustre: server umount lustre-MDT0000 complete [10426.124287] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [10427.356100] Key type lgssc unregistered [10427.504525] LNet: 474854:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10427.508594] LNet: Removed LNI 192.168.203.153@tcp [10427.894988] Key type .llcrypt unregistered [10427.896543] Key type ._llcrypt unregistered [10447.011660] alg: No test for adler32 (adler32-zlib) [10447.763382] Key type ._llcrypt registered [10447.764811] Key type .llcrypt registered [10447.823575] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [10447.968624] Key type .llcrypt unregistered [10447.970528] Key type ._llcrypt unregistered [10457.050659] alg: No test for adler32 (adler32-zlib) [10457.811414] Key type ._llcrypt registered [10457.812672] Key type .llcrypt registered [10457.859587] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10458.357324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10458.533698] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [10458.652176] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [10458.654486] LNet: Accept secure, port 988 [10460.263113] Key type lgssc registered [10460.809671] Lustre: Echo OBD driver; http://www.lustre.org/ [10464.824584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10468.438770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10471.751254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10475.675379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10475.727990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10476.862699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10476.878036] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10476.927274] Lustre: lustre-MDT0000: new disk, initializing [10476.997124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10477.009229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10479.405226] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10486.664145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10486.709382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10486.767840] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10486.793276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10486.797504] Lustre: Skipped 1 previous similar message [10486.845577] Lustre: lustre-MDT0001: new disk, initializing [10486.900677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10486.924232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10486.935813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10488.936506] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10494.068285] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10497.739253] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10501.394553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10501.439762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10501.443863] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10501.587829] Lustre: lustre-OST0000: new disk, initializing [10501.593238] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10501.641119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10504.157723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10504.163739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10504.247166] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10510.309989] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10514.402515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10514.404794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10514.411446] Lustre: Skipped 2 previous similar messages [10514.414475] Lustre: Skipped 1 previous similar message [10516.209865] Lustre: server umount lustre-MDT0000 complete [10519.020581] LustreError: 478338:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320706 with bad export cookie 15351477582023314599 [10519.022687] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10519.033219] LustreError: 478338:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10519.328174] Lustre: server umount lustre-MDT0001 complete [10528.224347] Lustre: 481591:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320709/real 1761320709] req@00000000cb68e7ff x1846878556666176/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761320715 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10528.378096] Lustre: server umount lustre-OST0000 complete [10535.362527] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 11:45:22 (1761320722) [10536.046465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10540.382449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10540.640130] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10540.688441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10543.203725] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10546.144383] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10547.948811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10548.143025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10550.725955] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10554.802978] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10557.999654] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10561.687027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10561.730506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10561.832374] Lustre: Found index 0 for lustre-OST0000, updating log [10561.931544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10564.133215] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10567.800519] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10571.176160] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [10576.353476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10576.354255] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10576.359920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10576.368650] Lustre: Skipped 2 previous similar messages [10577.330757] Lustre: server umount lustre-OST0000 complete [10581.474421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10581.478794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10581.481047] Lustre: Skipped 2 previous similar messages [10586.593491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10586.598592] Lustre: Skipped 2 previous similar messages [10588.794217] Lustre: server umount lustre-MDT0000 complete [10590.556379] LustreError: 482399:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320777 with bad export cookie 15351477582023315936 [10590.560175] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10590.561534] LustreError: 482399:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10594.722808] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 11:46:22 (1761320782) [10597.344106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10599.623381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10602.051121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10604.056232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10612.095675] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [10616.466575] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10621.700964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10625.346481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10629.314153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10632.631228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10636.466664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10636.510071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10636.671205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10636.702230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10636.747504] Lustre: lustre-MDT0000: new disk, initializing [10636.813761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10636.828053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10639.416365] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10645.830090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10645.882950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10645.938651] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10645.966566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10645.969320] Lustre: Skipped 1 previous similar message [10646.012694] Lustre: lustre-MDT0001: new disk, initializing [10646.092190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10646.100741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10648.423593] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10654.637818] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10657.728576] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10661.667339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10661.714666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10661.718699] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10661.850822] Lustre: lustre-OST0000: new disk, initializing [10661.852188] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [10661.853146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10661.855062] Lustre: Skipped 2 previous similar messages [10664.266894] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all 8 [10669.981889] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10670.132393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10670.139891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10672.763210] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10672.908230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10675.611598] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10675.734275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10677.217724] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10677.224301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10677.238416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10683.062112] Lustre: server umount lustre-OST0000 complete [10683.063926] Lustre: Skipped 1 previous similar message [10685.410605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10685.418198] Lustre: Skipped 2 previous similar messages [10693.586710] LustreError: 489424:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320880 with bad export cookie 15351477582023316993 [10693.588134] LustreError: 166-1: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10693.591851] LustreError: 489424:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10706.541888] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [10710.594444] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10715.714531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10719.856475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10722.928026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10726.224595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10727.010224] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10604 sec ========= 11:48:34 (1761320914) [10746.901250] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [10748.362612] Key type lgssc unregistered [10748.520681] LNet: 498156:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10748.526282] LNet: Removed LNI 192.168.203.153@tcp [10748.983837] Key type .llcrypt unregistered [10748.986854] Key type ._llcrypt unregistered